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(57) Abstract: In one embodiment, mirroring communi- 
cation from a second source storage device to a second tar- 
get storage device in a data mirroring system is disabled 
when mirroring communication from a first source storage 
device to a first target storage device is disabled. In another 
embodiment, information is stored in a data mirroring sys- 
tem identifying at least one subset of a plurality of mirrored 
source storage devices for which a consistent representation 
of data is desired to be maintained on at least one corre- 
sponding subset of a plurality of target storage devices. In 
yet another embodiment, at least one of a plurality of mir- 
rored source-storage devices is placed in one of a first state, 
a second state, and a third state. In response to a source stor- 
age system receiving a write requesting command chain de- 
tected to a mirrored source storage device in the first state, 
the command chain is permitted to commit and data writ- 
ten by the command chain is transferred to a corresponding 
target storage device. In response to the source storage sys- 
tem receiving a write-requesting command chain directed 
to a mirrored source storage device in the second state, the 
command chain is prevented from committing. In response 
to the source storage system receiving a write -requesting 
command chain directed to a mirrored source storage de- 
vice in the third state, the command chain is permitted to 
commit without transferring data written by the command 
chain to a corresponding target storage device. 
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METHOD \ND APPARATUS FOR MAINTAINING CONSISTENCY OF DATA 

STORED IN A GROUP OF MIRRORING DEVICES 

Field of the Invention 

The present invention is directed to a method and apparatus for maintaining 

consistency of data stored in a group of mirroring devices. 

Description of the Related Art 

A number of computer system configurations mirror information stored in source 
storage devices to target storage devices that can be disposed at locations remote from the 
source storage devices. An example of such a computer system is a data mirroring system 
100 discussed below in connection with Fig. 1 . 

In the data mirroring system 100 of Fig. 1, one or more host CPU's 102 are coupled 
(via one or more communication links 1 14) to a source storage system 1 04. In the example 
shown, the source storage system 104 includes a plurality of "source" storage devices 1 lOa-f. 
The source storage devices 1 lOa-f can be included in a single source storage system 104 as 
shown in Fig. 1 , or multiple source storage systems 1 04 can be employed, and the source 
storage devices 1 lOa-f can be distributed across the multiple source storage systems. Each 
source storage system 104 may, for example, comprise a storage system from the 
S YMMETRIX line of disc arrays available from EMC Corporation, Hopkinton, MA. 

The data stored in the source storage devices 1 10a- f may be crucial to the operation of 
the host CPU(s) 102. Therefore, a contingency solution is desirable in the event that a 
problem is experienced with any of the source storage devices 1 lOa-f to ensure that the data . 
stored thereby is not lost, and to minimize the risk of the host CPU(s) 1 02 being down due to 
problems with the source storage devices 1 10a- f. Potential problems with the source storage 
devices lT0a-f may include, for example, hardware or software errors that may make- stored 
data unrecoverable. Additionally, catastrophic events such as an earthquake or other natural 
disaster could result in the destruction of one or more of the source storage devices 1 10a- f. 

One solution for protecting the data stored in the source storage devices 1 lOa-f is to 
mirror the data in a corresponding set of "target" storage devices. The data mirroring system 
100 of Fig. 1 is an example of such a system, wherein the source storage system 104 is 
coupled (via links 1 12a-f) to a target storage system 106. The target storage system 106 
includes a plurality of target storage devices 1 1 6a- f -corresponding, respectively, to the source 
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storage devices 1 lOa-f in the source storage system 104. As data is written to any of the 
source storage devices 1 lOa-f, it is also written to a corresponding one of the target storage 
devices 1 16a-f in the target storage system 106. If one of the source storage devices 1 lOa-f is 
destroyed or experiences an error that renders stored data unrecoverable, the data can be 
retrieved from the corresponding one of the target storage devices 1 16a-f. The target storage 
devices 1 1 6a-f can be included in a single target storage system 1 06 as shown in Fig. 1 , or 
multiple target storage systems 106 can be employed, and the target storage devices 1 16a-f 
can be distributed across the multiple target storage systems 106. As with the source storage 
system(s) 104, each of the target storage systems 106 may, for example, be a storage system 
from the SYMMETRIX line of disc arrays available from EMC Corporation, Hopkinton, 
MA. 

When each of the source and target storage systems 104 and 106 is implemented using 
one or more of the SYMMETRIX line of disk arrays available from EMC Corporation, a 
feature called SYMMETRIX Remote Data Facility (SRDF) can be employed to implement 
the connection therebetween. SRDF is described in numerous publications available from 
EMC Corporation, including the SYMMETRIX Remote Data Facility Product Manual, P/N 
200-999-554, rev. B, June 1995. SRDF is also described in U.S. Patent No. 5,544,347 
(Yanai). 

In the illustrative data mirroring system 100 of Fig. 1, first, second, and third sets of 
source storage devices I lOa-b, 1 lOc-d, and 1 lOe-f, respectively, in the source storage system 
1 04 are in communication (via links 1 1 2a-b, 1 1 2c-d, and 1 1 2e-f, respectively) with 
corresponding first, second, and third sets of target storage devices 1 16a-b, 1 16c-d, and 1 16e-f 
in the target storage system 106. The links 1 12 are referred to herein as "mirroring links," and 
the term "mirroring communication" is used herein to refer to communication between source 
and target storage systems 104 and 106 over the mirroring links 1 12 that permits the target 
storage devices 1 16a-f to mirror data stored by the source storage devices 1 lOa-f. As shown 
in Fig. 1, the source storage devices 1 lOa-b, the mirroring links 1 12a-b, and the target storage 
devices 1 16a-b constitute a first mirroring group 108a; the source storage devices 1 lOc-d, the 
mirroring links 1 12c-d, and the target storage devices 1 16c-d constitute a second mirroring 
group 108b; and the source storage devices 1 lOe-f the mirroring links 1 12e-f, and the target 
storage devices 1 16e-f constitute a third mirroring group 108c. A "mirroring group" is a 
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eroup of storage devices in the source and target storage systems 104 and 106 between which 
data can be transferred via a common set of mirroring links 112. Generally, storage devices 
110 and 1 16 do not share any mirroring links 112 with storage devices 110 and 1 16 in other 

mirroring groups 108. 

To perform a write operation to the source storage system 104, a host CPU 102 
transmits an "I/O command chain" to the source storage system 104. Each I/O command 
chain can include one or more commands directed to a particular logical volume stored by the 
source storage system 104. A logical volume is a unit of information that the host CPU 102 
perceives to correspond to a single physical storage device, but that may be mapped within the 
source storage system 104 to physical storage space on one or more source storage devices 
110. An I/O command chain for a write operation ("a write-requesting command chain") 
includes various instructions for the source storage system 1 04 receiving the I/O command 
chain, as well as the data to be written during the operation. The host CPU 102 generally 
receives only a single message back from the source storage system 1 04 in response to its 
sending of a write-requesting command chain. This message generally indicates either: (1) 
that the I/O command chain has been "committed" (i.e., processed successfully) by the source 
storage system 1 04, or (2) that an error occurred preventing the I/O command chain from 
being committed by the source storage system 104. Typically, the source storage system 104 
will not provide the "I/O command chain committed" message back to the host CPU 102 
unless and until the data has been written successfully to one or more of the source storage 
devices 1 10, or to a temporary storage space (such as cache) in the source storage system 104. 

One of the purposes for mirroring data in the target storage system 106 is that if the 
data on the source storage system 104 is rendered unrecoverable, the data can be retrieved , 
from the target storage system 106. Thus, it is desirable to ensure that the data stored in the 
target storage system 106 is internally consistent, and reflects an accurate mirror of the data 
stored in the source storage system 104 at some particular point in time. If the data in the 
target storage system 106 does not represent an accurate mirror of the data in the source 
storage system 104 for a given point in time, the data in the target storage system 106 cannot 
be reloaded onto the source storage system 104 to place the source storage system 104 back 
into a valid state. For example, if the data stored in the source storage system 104 and 
mirrored on the target storage system 106 is included in a database, but the information stored 
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in target storage system 1 06 does not reflect an accurate picture of the database at a given 
point in time, then if an error is encountered that prevents the database from being recovered 
from the source storage system 104, the target storage system 106 cannot be used to recover 
the database because the data stored thereby is not an accurate reflection of the state of the 
database at any particular point in time. 

In a data mirroring system such as the data mirroring system 100 shown in Fig. 1, 
Applicant's have recognized that malfunctions with the mirroring links 1 12a-f that result in 
the loss of mirroring communication between the source and target storage systems 104 and 
106 for one of the mirroring groups 108a-c can cause a data consistency problem to arise 
under certain circumstances. Suppose, for example, that all of the mirroring links 1 12a-b for 
the mirroring group 108a were to malfunction so that mirroring communication over the 
mirroring links 1 12a-b was disabled, thereby preventing updates to the source storage devices 
1 10a- 1 10b from being transferred to the target storage devices 1 1 6a-b. Suppose further that 
mirroring operations were to continue normally over the mirroring links 1 12c-d and 1 12e-f for 
the mirroring groups 108b and 108c, respectively, so that updates to the source storage 
devices 1 lOc-f would continue to be transferred to the target storage devices 1 16c-f. If, under 
these circumstances, the data stored in either of the target storage devices 1 16a-b was 
logically related to the data stored in any of target storage devices 1 16c-f (e.g., if they stored 
potions of the same database), then the data stored in the target storage system 106 would not 
represent a valid state of the data stored in the source storage system 104 at any particular 
point in time. Therefore, if the source storage system 104 were to fail in the above-described 
situation, the data on the target storage system 106 could not be used to place the source 
storage system 104 back into a valid state for any particular point in time. 

The above-described problem may be better understood in light of the following 
illustrative situation in which the host CPU(s) 102 write "dependent" data units to respective 
source storage devices 110 when mirroring communication is disabled for only one of the 
source storage devices 1 1 0 to which the dependent data units are being written. A second 
unit of data is dependent on a first unit of data if the second unit of data is a function of the 
first unit of data so that, if the first unit of data is altered, the second unit of data must also be 
altered if it is to accurately reflect the current value of the first unit of data. For example, if 
first, second, and third units of data represent values "A," "B." and "A + B," respectively, 
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then the third unit of data is dependent on both the first and the second units of data, whereas 
the First and second units of data are not dependent on one another. 

Using this simple example of dependent data units, in the situation described above 
wherein mirroring communication is disabled over the mirroring links 1 12a-b but is enabled 
over the mirroring links 1 12c-f, if the host CPU(s) 102 initially write the first unit of data 
(i.e., the value t; A") to the source storage device 1 10a in the mirroring group 1 08a ("the 
disabled mirroring group"), and subsequently write the third unit of data (i.e., the value 
"A+B") to the source storage device 1 10c in the mirroring group 108b ("an operational 
mirroring group"), then both the first and third units of data would be written to the source 
storage system 104, but only the third unit of data (and not the first unit of data) would be 
written to the target storage system 106. As a result, the target storage system 106 would 
store an updated value of C4 A+B ? " without also storing an updated value of "A." This 
inconsistency can pose problems if it becomes necessary to recover the data from the target 
storage system 106. 

One technique that has been used in the past to avoid the above-described data 
inconsistency problem has been to prevent the source storage system 104 from committing 
any write-requesting command chain from a host CPU 102 to a source storage device 1 lOa-f 
if that source storage device 1 lOa-f is a member of a mirroring group 1 08 that is unable to 
engage in mirroring communication across its mirroring links 1 12. Using this technique, 
since each write-requesting command chain directed to a source storage device 1 10 in a 
disabled mirroring group 108 is not committed by the source storage system 104, the host 
CPU 102 transmitting the I/O command chain will repeatedly attempt (unsuccessfully) to 
perform the write operation. 

It should be appreciated that application programs executing on the host CPU(s) 102 
typically will not begin execution of an instruction that is dependent upon a unit of data 
updated by a previous instruction until the previous instruction completes, which requires that 
the I/O command chain that implements the previous instruction has been committed by the 
source storage system 104. Therefore, by preventing write-requesting command chains 
directed to source storage devices 1 lOa-f for which mirroring communication has been 
disabled from committing, it is ensured that no later instructions dependent on the data to be 
written by those write-requesting command claims will be executed,, thereby avoiding the 
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above-discussed data inconsistency problem. In this respect, the data stored in the target 
storage system 106 will be a valid mirror of the data stored in the source storage system 104 at 
the point that the mirroring group became disabled. 

It should be appreciated that although the above-described technique avoids a data 
inconsistency problem, it does so by preventing I/O command chains directed to source 
storage devices 1 1 0 within a disabled mirroring group from ever completing. As a result, the 
CPU 102 that initiates the I/O command chain will repeatedly attempt (unsuccessfully) to 
execute the command chain, eventually resulting in a crash of the application or other 
program that includes the command chain. 

Summary of the Invention 

According to one aspect of the present invention, a method for use in a data mirroring 
system involves disabling mirroring communication from a second source storage device to a 
second target storage device when mirroring communication from a first source storage device 
to a first target storage device is disabled. 

According to another aspect of the present invention, a source storage system for use 
in a data mirroring system includes at least first and second source storage devices to be 
coupled to at least one host to enable the at least one host to perform write operations to the 
first and second source storage devices, and further to be coupled to at least first and second 
target storage devices, respectively, to enable mirroring communication from the at least first 
and second source storage devices to the at least first and second target storage devices, and at 
least one controller to, responsive to mirroring communication from the first source storage 
device to the first target storage device becoming disabled, disable mirroring communication 
from the second source storage device to the second target storage device. 

According to another aspect of the invention, at least one computer-readable medium 
has a plurality of instructions stored thereon which, when executed by at least one processor 
included in a data mirroring system, cause the at least one processor perform a method 
including a step of disabling mirroring communication from a second source storage device to 
a second target storage device when mirroring communication from a first source storage 
device to a first target storage device is disabled. 

According to another aspect of the present invention, a method for use in a data 
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mirroring system involves instructing at least one source storage system to disable mirroring 
communication from a second source storage device to a second target storage device in 
response to at least one host receiving an indication that mirroring communication from a first 
source storage device to a first target storage device is disabled. 

According to another aspect of the invention, at least one computer-readable medium 
has a plurality of instructions stored thereon which, when executed by at least one host 
processor included in a data mirroring system, cause the at least one host processor to perform 
a method including a step of instructing the at least one source storage system to disable 
mirroring communication from a second source storage device to a second target storage 
device in response to the at least one host processor receiving an indication that mirroring 
communication from a first source storage device to a first target storage device is disabled. 

According to another aspect, a host for use in a data mirroring system includes at least 
one controller to be coupled to at least first and second source storage devices to perform 
write operations to the at least first and second source storage devices. The at least one 
controller is configured to, responsive to receiving an indication that mirroring 
communication from the first source storage device to a first target storage device is disabled, 
instruct at least one source storage system in which the at least first and second source storage 
devices are included to disable mirroring communication from the second source storage 
device to a second target storage device. 

According to another aspect of the invention, a method for use in a data mirroring 
system involves storing information in the data mirroring system identifying at least one 
subset "of a plurality of mirrored source storage devices for which a consistent representation 
of data is desired to be maintained on at least one corresponding subset of a plurality of target 
storage devices. 

According to another aspect of the invention, a host for use in a data mirroring system 
includes at least one memory, and at least one controller to store information in the at least 
one memory identifying at least one subset of a plurality of mirrored source storage devices 
for which a consistent representation of data is desired to be maintained on at least one 
corresponding subset of a plurality of target storage devices. 

According to another aspect of the invention, a source storage system for use in a data 
mirroring system includes at least one memory, and at least one controller to store information 
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in the at least one memory identifying at least one subset of a plurality of mirrored source 
storage devices for which a consistent representation of data is desired to be maintained on at 
least one corresponding subset of a plurality of target storage devices. 

According to another aspect of the invention, a method for use in a data mirroring 
system involves placing at least one of a plurality of mirrored source storage devices in one of 
a first state, a second state, and a third state. In response to a source storage system receiving 
a write-requesting command chain directed to the at least one of the plurality of mirrored 
source storage devices when the at least one of the plurality of mirrored source storage 
devices is in the first state, the command chain is permitted to commit and data written by the 
command chain is transferred to a corresponding one of a plurality of target storage devices. 
In response to the source storage system receiving a write-requesting command chain directed 
to the at least one of the plurality of mirrored source storage devices when the at least one of 
the plurality of mirrored source storage devices is in the second state, the command chain is 
prevented from committing. In response to the source storage system receiving a write- 
requesting command chain directed to the at least one of the plurality of mirrored source 
storage devices when the at least one of the plurality of mirrored source storage devices is in 
the third state, the command chain is permitted to commit without transferring data written by 
the command chain to the corresponding one of the plurality of target storage devices. 

According to yet another aspect of the invention, a source storage system for use in a 
data mirroring system includes a plurality of mirrored source storage devices to be coupled to 
at least one host to permit the at least one host to perform write operations to the plurality of 
mirrored source storage devices, and further coupled to a plurality of target storage devices to 
enable mirroring communication from each of the plurality of mirrored source storage devices 
to a corresponding one of the plurality of target storage devices, and at least one controller to 
place at least one of the plurality of mirrored source storage devices in one of a first state, a 
second state, and a third state, wherein, in response to the source storage system receiving a 
write-requesting command chain directed to the at least one of the plurality of mirrored source 
storage devices when the at least one of the plurality of mirrored source storage devices is in 
the first state, the at least one controller permits the command chain to commit and transfers 
data written by the command chain to the corresponding one of the plurality of target storage 
devices, wherein, in response to the source storage system receiving a write-requesting 
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command chain directed to the at Least one of the plurality of mirrored source storage devices 
when the at least one of the plurality of mirrored source storage devices is in the second state, 
the at least one controller prevents the command chain from committing, and wherein, in 
response to the source storage system receiving a write-requesting command chain directed to 
the at least one of the plurality of mirrored source storage devices when the at least one of the 
plurality of mirrored source storage devices is in the third state, the at least one controller 
permits the command chain to commit without transferring data written by the command 
chain to the corresponding one of the plurality of target storage devices. 

According to another aspect of the invention, at least one computer-readable has a 
plurality of instructions stored thereon which, when executed by at least one processor 
included in a data mirroring system, cause the at least one processor to perform a method 
including steps of: (A) placing at least one of a plurality of mirrored source storage devices in 
one of a first state, a second state, and a third state; (Bl) in response to a source storage 
system receiving a write-requesting command chain directed to the at least one of the plurality 
of mirrored source storage devices when the at least one of the plurality of mirrored source 
storage devices is in the first state, permitting the command chain to commit and transferring 
data written by the command chain to a corresponding one of a plurality of target storage 
devices; (B2) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the second state, . 
preventing the command chain from committing; and (B3) in response to the source storage 
system receiving a write-requesting command chain directed to the at least one of the plurality 
of mirrored source storage devices when the at least one of the plurality of mirrored source 
storage devices is in the third state, permitting the command chain to commit without 
transferring data written by the command chain to a corresponding one of the plurality of 
target storage devices. . 

Brief Description of the Drawings 

Fig.l is a block diagram showing an illustrative example of a data mirroring system 
with which aspects of the present inventions can be employed; 

Fig. 2 is a block diagram showing an example of a data mirroring system such as that 
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shown in Fig. 1 that employs multiple source and target storage systems; 

Fig. 3 is a block diagram showing one illustrative implementation of a storage system 
that can be used as the source and/or target storage systems of Figs. 1 and 2; 

Figs. 4A-B and 5 illustrative examples of tables of flags that may be stored in each of 
the source storage systems of Figs. 1-2 to assist in maintaining the consistency of data stored 
on a plurality of target storage devices according to one embodiment of the present invention; 

Fig. 6 is a flow chart showing an illustrative example of a process that may performed 
by one or more controllers of each of the source storage systems of Figs. 1 and 2 to implement 
various aspects of the present invention relating to maintaining the consistency of data stored 
on a plurality of target storage devices; and 

Fig. 7 is a flow chart showing an illustrative example of a software routine that may be 
performed by the host CPUs of Figs. 1 and 2 to implement additional aspects of the present 
invention relating to maintaining the consistency of data stored on a plurality of target storage 
devices. 

Detailed Description of the Invention 

According to one aspect of the present invention, in a data mirroring system such as 
the data mirroring system 100 of Fig. 1, steps are taken to ensure that a consistent data set is 
maintained on the target storage system 106 when mirroring communication for a source 
storage device 1 10 is lost, without also causing the software process (e.g., an application 
program) on the host CPU(s) 102 to crash. W-d^*^ 

It should be appreciated that the above-described data inconsistency problem that can 
result from the loss of mirroring communication is attributable to the fact that the mirroring 
links 1 12a-f are operational for some of the mirroring groups 108a-c, but not for all of them. 
In accordance with one illustrative embodiment of the present invention, when it is 
determined that all of the mirroring links 1 12a-f for a particular mirroring group 108 have 
become disabled (i.e., no longer support mirroring communication between the source and 
target storage devices 1 10 and 1 16 in the mirroring group 108), mirroring communication 
over the mirroring links 1 12a-f for the other mirroring groups 108 may also be disabled. This 
can be considered as "logically breaking" the mirroring links 1 12a-f associated with these 
other mirroring groups 108. As such, it should be appreciated that breaking all of the 
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mirroring links 1 12a-f at the same time (e.g., a time when the mirroring links 1 12a-f are 
physically disabled for a particular mirroring group 108) prevents any further updates from 
being made to any of the target storage devices 1 16a-f. Thus, after all of the mirroring links 
1 1 Oa-f have been broken, the target storage system 106 stores a set of data that represents an 
accurate mirror of data stored in the source storage system 1 04 at a particular point in time 
(i.e., the point in time when mirroring communication for one of the mirroring groups 108 
became disabled). 

Although logically breaking the remaining mirroring links 1 12a-f will result in a loss 
of the mirroring capability by the target storage system 106, it should be appreciated that after 
all of the mirroring links 1 12a-f have been logically broken, the source storage system 104 
can asain commit write-requesting command chains directed to the source storage devices 
1 10 included in the disabled mirroring group 108, and the application programs executing on 
the host CPU(s) 102 can continue operating without crashing as a result of repeatedly 
attempting (unsuccessfully) to perform such write operations. Meanwhile, the problem 
experienced by the mirroring links 112 that resulted in the loss of mirroring communication 
for one of the mirroring groups 108 can be addressed. Once corrected, all of the mirroring 
links 1 12a-f can be restored, thereby enabling the mirroring capability provided by the target 
storage system 106 to be brought back on line. 

It should be appreciated that logically breaking all of the mirroring links 1 12a-f 
whenever the mirroring links 1 12 are lost for a particular mirroring group 108 achieves the 
result of preventing the above-discussed data consistency problem for the data on the target 
storage system 106. This can be accomplished according to one embodiment of the present 
invention in the manner discussed below. However, applicants have also recognized that it 
may not be necessary to logically break all of the mirroring links 1 12a-f to achieve the data 
consistency goals discussed above. In this respect, it should be appreciated that the target 
storage system 106 may include multiple sets of logically unrelated data, e.g., multiple 
unrelated databases. For such logically unrelated sets of data, it is unnecessary to maintain 
consistency therebetween. Thus, if a loss were experienced of the mirroring links 1 12a-f 
associated with a mirroring group 108 including target storage devices 116 that store a 
particular database, it would be unnecessary to logically break the mirroring links 112 
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associated with other mirroring groups 108 including only target storage devices 116 that 
store unrelated data. Therefore, in accordance with another illustrative embodiment of the 
invention, the mirroring links 1 12a-f are logically broken only for mirroring groups 108 that 
include target storage devices 116 that store data logically related to the data stored in the 
target storage devices 1 1 6 associated with a mirroring group 1 08 for which mirroring 
communication has been lost. In this manner, the impact on the mirroring capabilities of the 
target storage system 106 are minimized. 

Similarly, it should be appreciated that all of the target storage devices 1 16a-f in a 
particular mirroring group 108a-c may not store logically related data. Thus, in another 
illustrative embodiment of the invention, mirroring links 1 12 within a mirroring group 108 
may be logically broken to disable mirroring communication between only some of the source 
and target storage devices 1 10 and 1 16 that the mirroring links 1 12 support. For example, if 
target storage devices 1 1 6a-c all store data for a common database, and the mirroring links 
1 12a-b for minoring group 108a become disabled, then the mirroring links 1 1 2c-d for 
mirroring group 108b can be logically broken to disable mirroring communication between 
the source storage device 1 10c and the target storage device 1 16c to maintain consistency of 
the data in the target storage system 106 for the database, without also disabling mirroring 
communication between the source storage device 1 1 Od and the target storage device 1 1 6d. 
Therefore, if the source and target storage devices 1 1 Od and 1 1 6d store data that is unrelated 
to the database stored by the source and target storage devices 1 lOa-c and 1 I6a-c, then 
mirroring communication over the mirroring links 1 12c-d of mirroring group 108b may 
remain enabled between source storage device 1 1 Od and target storage device 1 1 6d, thereby 
minimizing the impact on the mirroring capabilities of the target storage system 106. 

The above-described embodiments of the present invention can be implemented in 
numerous ways, and the present invention is not limited to any particular implementation. In 
accordance with the above-identified embodiments of the invention wherein mirroring 
communication across mirroring links 1 12a-f is selectively disabled on a device-by-device 
basis, the source storage deuces 1 lOa-f may, for example, be categorized into one or more so- 
called "consistency groups » A consistency group in this context is a group of source storage 
devices 1 10 for which an effort is made to ensure that the data stored therein and in the 
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corresponding target storage devices 1 16 is kept consistent. For example, if the source 
storage devices 1 10a and 1 10c store dependent units of data, it may be desirable to include the 
source storage devices 1 1 0a and 1 1 0c in the same consistency group so that efforts can be 
made to prevent data consistency problems such as those described above. 

Consistency groups can be implemented in any of numerous ways, and the present 
invention is not limited to any particular implementation. In one embodiment, for example, 
the source storage system 104 stores information identifying only whether or not each source 
storage device 1 10 is a member of a consistency group, and information identifying the 
particular source storage devices 1 10 in each consistency group is maintained only by the host 
CPU(s) 102. It should be appreciated, however, that such consistency group information can 
be stored in numerous other ways, as the invention is not limited to the embodiment wherein 
most consistency group information is maintained by the host CPU(s) 102. For example, all 
relevant consistency group information may alternatively be stored in memory associated with 
the source and/or target storage systems 104 and 106. 

In one embodiment of the invention, when a host CPU 102 (Fig. 1) issues a write- 
requesting command chain to one of the source storage devices 1 1 Oa-f (the "destination- 
storage device), as long as at least one of the mirroring links 1 12 in the mirroring group 108 
of the destination storage device 1 10 is operational (i.e., mirroring communication for the 
destination storage device 1 10 is enabled), the source storage system 104 will commit the I/O 
command chain after the to-be-written data has been transferred to both the source storage 
system 104 and the target storage system 106. When all of the mirroring links 1 12 for a 
mirroring group 108 become non-operational (i.e., mirroring communication for the 
destination storage device 110 is disabled), this is detected when a host CPU 102 attempts to 
-write to a- source storage device 1 10 in the mirroring group 108. Specifically, when a host 
CPU 1 02 issues a write-requesting command chain to a destination storage device 1 1 0 for 
which mirroring communication is disabled, the source storage system 104 including the 
destination storage device 1 1 0 temporarily prevents write-requesting command chains 
directed to the destination storage device 1 10 from committing. 

During the time period that write-requesting command chains directed to the 
destination storage device 1 10 are temporarily prevented from committing, steps may be taken 



WO 01/16748 - 14 - PCT/US00/24128 

to prevent all of the other source storage devices 1 lOa-f that are included in the same 
consistency group as the destination storage device 1 10 from communicating across the 
mirroring links 112 associated therewith (i.e., mirroring communication for these source 
storage devices 110 is disabled). This disabling of mirroring communication for the relevant 
source storage devices 1 10 can be accomplished in numerous ways, and the invention is not 
limited to any particular implementation. For example, flags may be stored in a memory 
associated with the source storage devices 110 indicating that certain source storage devices 
1 10 are not to transfer data across the mirroring links 1 12. Alternatively, the relevant 
mirroring links 1 12 may be physically broken. However, because a single mirroring group 
108 may contain source storage devices 1 10 from several consistency groups, as discussed 
above, it may be desirable to disable mirroring communication across the mirroring links 1 12 
on a device-by-device basis, so that information can continue to be transferred across the 
mirroring links 1 12 for the source storage devices 1 10 that are not included in the relevant 
consistency group. 

After mirroring communication across the mirroring links 1 12 has been disabled for 
each of the source storage devices 1 10 included in the same consistency group as the 
destination storage device 110, the source storage system 104 including the destination 
storage device 1 10 is again permitted to commit the write-requesting command chain directed 
to the destination storage device 110. Because mirroring communication has been disabled 
for all source storage devices 1 10 in the destination storage device's consistency group, the 
target storage devices 1 16a-f in this consistency group will store a set of data representing a 
"snapshot" of the . data stored by the source storage system 104 just prior to the time that 
mirroring communication was disabled for the consistency group. In accordance with this 
embodiment of the invention, because the source storage system 104 only temporarily 
prevents write-requesting command chains directed to source storage devices 1 10 included in 
disabled mirroring groups 108 from committing, data consistency problems such as those 
discussed above can be avoided without causing the application software running on the host 
CPU(s) 102 to crash. 

Applicants have recognized that, when disabling mirroring communication across 
mirroring links 112 for multiple source and target storage devices 1 10 and 1 1 6 in response to 
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mirroring communication for a mirroring group 108 being lost, the disabling of mirroring 
communication for different ones of these multiple source storage devices 1 10 at different 
times can itself cause a data consistency problem similar to that described above. For 
example, assume that source storage devices 1 10a, 1 10c,' and 1 lOe of Fig. 1 are all included in 
the same consistency group, and mirroring communication is to be disabled for source storage 
devices 1 10c and 1 lOe in response to the detection, when trying to write to storage device 
1 10a. that mirroring communication for source storage device 1 10a has been lost. If the 
source storage system 1 04 is permitted to commit write-requesting command chains directed 
to either of the source storage devices 1 10c and lOOe prior to verifying that mirroring 
communication for both devices has been disabled, the relative timing of the disabling of 
mirroring communication for the source storage devices 1 10c and 1 lOe may result in the 
occurrence of a data consistency problem in the target storage system 106. For example, if 
mirroring communication was disabled for the source storage device 1 10c before source 
storage device 1 lOe, a data consistency problem would arise if an I/O command chain 
requesting the write of a first unit of data to the source storage device 1 10c was committed by 
the source storage system 104 after mirroring communication for the source storage device 
1 10c was disabled, and then a subsequent I/O command chain requesting the write of a second 
unit of data, dependent on the first unit of data, to the source storage device 1 lOe was both 
committed by the source storage system 104 and transferred to the target storage system 106 
before mirroring communication for the source storage device 1 lOe was disabled. 

To address this potential problem, in one embodiment of the invention, when 
mirroring communication across any of the mirroring links 1 12 is to be disabled for source 
storage devices 1 10 within a consistency group, the source storage system 104 is temporarily 
prevented from committing- write-requesting command chains directed to all source storage 
devices 1 10 in the consistency group. Only after mirroring communication for all of the 
source storage devices 1 10 in the relevant consistency group has been disabled is the source 
storage system 104 again permitted to commit write-requesting command chains directed to 
these source storage devices 110. In this manner, the host CPU(s) 102 will be unable to 
successfully write to these source storage devices 1 10 only during the period of time that 
mirroring communication is being disabled for all of the source storage devices 1 1 0 in the 
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relevant consistency group. 

The above-described technique for selectively disabling mirroring communication for 
certain source storage devices 110 can be accomplished in any of a number of ways, and the 
invention is not limited to any particular implementation. One illustrative example of a 
technique that may be used to achieve these objectives is described below in connection with 
Figs. 2-7, wherein certain flags stored in each source storage system 104 indicate how that 
source storage system 104 should respond to write-requesting command chains from the host 
CPU(s) 102 that are directed to its source storage devices 110. It should be appreciated, 
however, that other techniques and/or different flags than those described below may 
alternatively be employed, and that the invention is not limited to the particular technique or 
flags described. 

Fig. 2 shows an illustrative example of a data mirroring system such as that shown in 
Fig. 1, including multiple source storage systems 104a-b and multiple target storage systems 
106a-b. In the embodiment shown, the host CPUs 102 include a pair of host CPUs 102a-b. 
As shown, each of the host CPUs 102a and 102b may be coupled (via communication links 
1 14a-d) to both of the source storage systems 104a and 104b, and each of the source storage 
systems 104a and 104b may be coupled (via the mirroring links 1 12a-f) to one or both of the 
target storage systems 106a and 106b. If the mirroring groups 108a-c of Fig. 1 were 
implemented on the data mirroring system of Fig. 2, the mirroring group 108a (Fig. 1) would 
be supported by the source storage system 104a and the target storage system 106a, the 
mirroring group 108b (Fig. 1) would be supported by the source storage system 104a and the 
target storage system 106b, and the mirroring group 108c (Fig. 1) would be supported by the 
source storage system 104b and the target storage system 106b. It should be appreciated that 
Figr2 shows only one example of how multiple CPU(-s) 1-02- and multiple storage systems 104 
and 106 may be combined in a manner supporting a plurality of mirroring groups, and that 
any of numerous' alternative configurations may instead be employed. 

Fig. 3 shows an illustrative example of a disc drive storage system that may be used in 
the data mirroring system 100 of Fig. 1 as the source storage system 104 and/or the target 
storage system 106. It should be appreciated that the source storage system 104 and/or the 
target storage system 106 of Fig. 1 may each employ a single storage system such as that 
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shown in Fig. 3, or several such storage systems working in concert. It should also be 
appreciated that the present invention need not employ storage systems of the type shown in 
Fig. 3, and that any of numerous other types of storage systems (e.g., tape drive systems) can 
alternatively be employed. 

As shown, the illustrative storage system 104b in Fig. 3 includes a host bus controller 
302, a plurality of disc controllers 304a-b, and a plurality of disc drives 3l2a-b. In the 
example shown, the disc drives 31 2a-b correspond to the source storage devices 1 lOe-f of 
Figs. 1 -2. The host bus controller 302 is responsible for communicating with the host CPUs 
102a-b (Fig. 2) via the communication links 1 14b and 1 14d. Each of the disc controllers 
304a-b is responsible for controlling a subset of the disc drives 3 12a-b. In the example 
system of Fig. 3, a mirroring controller 308 controls mirroring communication for the disc 
drives 312a-b. For example, when the storage system 104a is used to implement the source 
storage system 104 of Figs. 2-3, the mirroring controller 308 may control mirroring 
communication across the mirroring links 1 12e-f for the mirroring group 108c. It should be 
appreciated that in a computer system such as that shown in Figs. 1-3, each of the storage 
systems 104 and 106 that supports more than one mirroring group 108 may, to facilitate this 
implementation, include a separate mirroring controller 308 for each mirroring group 108 that 
it supports. 

In the Fig. 3 embodiment, the controllers 302, 308 and 304a-b communicate via an 
internal bus 310. Also connected to the bus 310 is a memory unit 306 including a global 
memory 306a and a cache 306b. The global memory 306a stores information that facilitates 
operation and control of the source storage system 104a. The cache 306b provides for 
improved system performance. In particular, when one of the host CPUs 102a-b (Fig. 2) 
executes a read of a track of information from the storage system 1 04b, if the track of 
information is already stored in the cache 306b, the source storage system 104b can service 
the read from the cache 306b, rather than from one of the disc drives 3 12a-b, and can thereby 
execute the read more efficiently. Similarly, when one of the CPUs 1 02a-b executes a write 
to the source storage system 104b, the host bus controller 302 typically executes the write to 
the cache 306b, and causes the I/O command chain requesting the write to be committed as 
soon as the data is stored in the cache 306b. Thereafter, the write may be destaged (in a 
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manner transparent to the host CPUs 102a-b) to the appropriate one of the disc drives 312a-b. 
The cache can therefore be viewed as a temporary storage device, and each of the disc drives 
310a-b can be viewed as a permanent storage device within the storage system 104b. 

With regard to the embodiment of Fig. 3, it should be appreciated that fewer or 
additional controllers than those shown can be used to perform the functionality of the various 
embodiments of the invention, and that the invention is not limited to any particular type or 
number of controllers. In different embodiments, for example, the functionality of all of the 
controllers 302, 308 and 304a-b of Fig. 3 may be implemented by a single controller, or the 
functionality of any one of the controllers 302, 308 and 304a-b may be implemented by 
another one of the controllers 302, 308 and 304a-b, or by one or more additional controllers 
(not shown). As used herein, the term "controller" refers to any combination of hardware 
(e.g., one or more processors and/or dedicated circuit components), software and/or firmware 
that is capable of performing one or more of the control functions described herein. 

Figs. 4A-B and 5 show examples of tables of flags 402A-B and 504 that may be stored 
for each source storage system 104 to permit the source storage systems 104 to ensure, in 
accordance with one embodiment of the present invention, that the target storage systems 106 
associated therewith do not encounter data consistency errors when mirroring communication 
between the source and target storage systems 104 and 106 is disabled for one or more 
mirroring groups 108. An illustrative method by which a source storage system 104 may use 
the information of Figs. 4 and 5 to achieve this result is discussed below in connection with 
Fig. 6. 

As shown in Figs. 4A-B, for each mirroring group 108 serviced by a source storage 
system 104, a respective table of flags 402 may be stored. Each table 402 reflects the current 
status- (i.e.,- enabled- sr- disabled) of each of the mirroring links 112 included in the mirroring 
group 108 with which it is associated. In the example of Figs. 4A-B, the tables 402 for two 
Mirroring groups (i.e., mirroring groups 108a and 108b) are shown, meaning that the source 
storage system 104 for which the information of Figs. 4A-B is stored services two distinct 
mirroring groups 108. The tables 402 A-B can be stored in any of numerous locations 
accessible to the source storage system 104 that uses them, and the invention does not require 
) that the tables 402A-B be stored in any particular location. In one illustrative embodiment, 
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for example, the tables 402A-B are stored in the global memory 306a of the source storage 

system 104a of Fig. 2. 

A mirroring link 1 12 is "disabled" for a given source-target storage device pair when 
the mirroring link 1 12 is unable to transfer information from the source storage device 1 10 to 
the target storage device 1 16. In the illustrative embodiment of the invention that employs the 
tables 402 A-B, the "enabled" and "disabled" entries in the tables 402A-B indicate the 
physical status of the mirroring links 1 1 2, i.e., whether the links 1 1 2 are operational or 
physically broken. Thus, a "disabled" entry in one of the tables 402A-B indicates that the 
corresponding mirroring link is physically broken so that no data can be transferred over the 
corresponding mirroring link 1 1 2 between any two storage devices 110,116, whereas an 
"enabled" entry indicates that the corresponding mirroring link 1 12 is operational so that data 
can be transferred over the mirroring link 1 12 between any source-target storage device pair 

associated therewith. 

As mentioned above, each mirroring group 108 may have a separate mirroring 
controller 308 associated with it. Because each mirroring controller 308 is in direct 
communication with the mirroring links 1 12 in its associated mirroring group 108, the entries 
in the tables 402 A-B in the illustrative embodiment shown may be updated only by the 
mirroring controllers 308 associated with the mirroring links 1 12 for the respective mirroring 
aroups 108. However, the present invention is not limited in this respect, as any device 
havina knowledge about the status of the mirroring links 1 12 and access to the memory in 
which the tables 402A-B are stored can update the tables 402 as discussed below. In addition 
to the mirroring controller 308, other devices having access to the global memory 306a (e.g., 
the host bus controller 302 (Fig. 3) of the source storage system 1 04) may also utilize the 
information stored in the tables 402 A-B in accordance with one embodiment of the invention 
discussed below. 

Fig. 5 illustrates another table of flags 504 which may be stored for each source 
storage system 104. The flags 504 reflect, for each storage device 1 lO.in the source storage 
system 104, (a) whether the storage device is included in a consistency group, and (b) the 
current "state" of the storage device. The table 504 may, for example, be stored in the global 
) memory 306a of the source storage system 104a of Fig. 2. It should be appreciated, however, 
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that the invention is not limited in this respect, and that the table 504 may alternatively be 
stored in any other location accessible to the source storage system 104. According to the 
illustrative embodiment shown, the flags in the table 504 identify each storage device 110 as 
being in one of three states: (1) "ON-LINE," (2) "OFF-LINE or (3) "PENDING OFF- 
LINE." The meaning and significance of each of these states is described below in connection 
with the illustrative software routines of Figs. 6 and 7. 

Fig. 6 shows an example of a software routine 600 that can be performed by one or 
more controllers (e.g., the host bus controller 302 of Fig. 3) in each of the source storage 
systems 104 to implement an embodiment of the invention that uses the table entries of Figs. 
4 and 5 to maintain the consistency of the data stored on the target storage system(s) 106. The 
software routine 600 may include a plurality of instructions stored in a computer readable 
medium, and the controller(s) implementing the routine 600 may, for example, execute the 
instructions stored on the computer-readable medium to perform the various method steps 
shown. Each of the controllers (e.g., 302 and 304 in Fig. 3) in the source storage system 104 
may include a processor having a local memory in which microcode instructions to implement 
the routine 600 can be stored, or the routine 600 can be provided in another memory (e.g., 
global memory 306) accessible to the controller(s). Alternatively, of course, the routine 600 
Li be implemented using dedicated hardware, or any combination of hardware and software 
capable of achieving a similar result. With regard to the illustrative routine 600 of Fig. 6, it 
should be appreciated that the precise order of the method steps is not critical, and that the 
invention is not limited to embodiments that perform method steps precisely in the order 
shown. Additionally, it should be appreciated that the method steps shown in Fig. 6 represent 
only one of numerous possible routines that can achieve the desired result, and the invention 
is not limited to the particular routine shown. Further, it should be understood that some 
embodiments of the invention can perform fewer than all of the functions performed by the 
method steps illustrated in Fig. 6, and the invention is not limited to embodiments that employ 
all of the functions performed by the illustrated routine. 

The routine 600 begins when the source storage system 104 (e.g., the source storage 
system 104b of Fig. 3) has received an I/O command chain from a host CPU 102 via a 
communication link 114. When an I/O command chain is received, it is determined (at step 
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604) whether the I/O command chain is requesting that a write be performed to one of the 
source storage system's storage devices 110 (e.g., one of the disc drives 312a-b of Fig. 3). 

If the I/O command chain is not requesting that a write be performed (e.g., if it is 
requesting that a read be performed), then the routine 600 proceeds to step 606, wherein the 
5 I/O command chain is permitted to be processed and committed by the source storage system 
104 in the normal manner, and the routine 600 terminates. Because I/O command chains that 
do not perform writes do not require the transmission of newly written data to a target storage 
system 106 over mirroring links 1 12, the status of the mirroring links 1 12 is not relevant to 
these operations, and such I/O command chains are processed and committed regardless of the 

1 0 status of the mirroring links 112. 

If it is determined (at the step 604) that the I/O command chain is requesting that a 
write be performed, then the routine 600 proceeds to steps 608-610, wherein the state flag (see 
Fig. 5) for the destination storage device 1 10 (i.e., the source storage device 1 10 to which the 
write-requesting command chain is directed) is checked to determine whether the destination 

15 storage device 1 10 is OFF-LINE (step 608), PENDING OFF-LINE (step 610), or ON-LINE 
(which is assumed if the step 610 determines that the destination storage device is not 

PENDING OFF-LINE). 

If (at the step 608) it is determined that the destination storage device 1 1 0 is OFF- 
LINE, then the routine 600 proceeds to the step 606, wherein the write-requesting command 
20 chain is permitted to be processed and committed by the source storage system 1 04. Because 
the destination storage device 1 10 is already in the OFF-LINE state, the routine 600 
recognizes that the destination storage device 1 1 0 either (1) does not have a corresponding 
target storage device 1 1 6 associated with it, (2) was turned OFF-LINE in response to a 
previous iteration of the routine 600 (e.g., in step 620 or step 626), or (3) was turned OFF- 
25 LINE by an iteration of a routine 700 (Fig. 7) by a host CPU 102 (e.g., in step 712 of the 

routine 700) as discussed below. In the latter two cases, the previous iteration of the routine 
600 or 700 will have performed steps (discussed below) to ensure that committing the write- 
requesting command chain at the present time, without also transferring the written data to the 
target storage system 106, will not result in a data consistency problem in the target storage 
30 system 106. 
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If (at the step 610) it is determined that the destination storage device 1 10 is 
PENDING OFF-LINE, the routine 600 proceeds to the step 612, wherein the source storage 
system 104 does not permit the write-requesting command chain to commit, and transmits a 
"special" I/O failure message back to the host CPU 102 that issued the I/O command chain. 
The routine 600 then terminates. The special I/O failure message indicates that the destination 
storage device 1 10 is in the PENDING OFF-LINE state. The destination storage device 110 
may, for example, have been placed in the PENDING OFF-LINE state during a previous 
iteration of the routine 600 (e.g., in step 622 or step 628), or during an iteration of the routine 
700 (Fig. 7) by a host CPU 102 (e.g., in step 708 of the routine 700) discussed below. In one 
illustrative embodiment, the special I/O failure message is selected so that it does not 
correspond to any other I/O failure message generated by source storage system 104. 
Therefore, the host CPU(s) 102 will always recognize the special I/O failure message as being 
uniquely indicative of an I/O failure resulting from a write-requesting command chain being 
directed to a source storage device 1 10 in the PENDING OFF-LINE state. 

When a host CPU 102 receives the special I/O failure message, it responds by taking 
steps to place all of the source storage devices 1 lOa-f that are included in the destination 
storage device's consistency group in the PENDING OFF-LINE state (see steps 706 and 708 
in the illustrative routine 700 of Fig. 7). After all source storage devices 1 lOa-f in the relevant 
consistency group have been placed in the PENDING OFF-LINE state, any host CPUs 102 
that received the special I/O failure message will then take steps (e.g., according to the routine 
700 described below in connection with Fig. 7) to place all of the source storage devices 
1 10a- f included in the relevant consistency group in the OFF-LINE state (see steps 710 and 
712 of the illustrative routine 700 of Fig. 7). In this regard, it should be appreciated that 
" several different host CPUs 102 may attempt to. write to source storage devices 110 that are in 
the PENDING OFF-LINE state before these device are subsequently turned to the OFF-LINE 
state. In such a situation, each host CPU 102 that attempted to write to a source storage 
device 1 10 in the PENDING OFF-LINE state will perform the routine 700 discussed below. 
Therefore, multiple host CPUs 102 may each independently be performing the routine 700 at 
the same time. After each of the storage devices 104 has been placed in the OFF-LINE state, 
the source storage system 104 is again permitted to commit write-requesting command chains 
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directed to that storage device 1 10 (see step 608 of routine 600). 

Thus, the PENDING OFF-LINE state of a destination storage device 1 10 corresponds 
to the situation described above wherein a source storage system 1 04 temporarily does not 
permit write-requesting command chains directed to the destination storage device 1 10 to 
comrnit during the time period that mirroring communication is being disabled for all other 
source storage devices 1 10 in the destination storage device's consistency group. The OFF- 
LINE state of a destination storage device 1 10, on the other hand, corresponds to the situation 
described above wherein a source storage system 104 permits write-requesting command 
chains to commit to a destination storage device 1 10 without first requiring the written data to 
be transferred to the target storage system 106. 

It should be appreciated that any host CPU 102 that receives the special error message 
in response to an attempted write to a destination storage device 110 will not immediately 
repeat attempts to perform the write and resultingly cause the application programs running on 
the host CPU 102 to crash. Rather, each such host CPU 102 will instead perform the routine 
700 (Fig. 7) to first place all source storage devices 1 10 in the relevant consistency group in 
the PENDING OFF-LINE state, and then place all such source storage devices 1 10 in the 
OFF-LINE state. After each host CPU 102 that receives the special error message has 
completed the routine 700, it will then repeat its attempt to perform the write operation. After 
the routine 700 has completed, all source storage devices 1 10 in the destination storage 
device's consistency group will be OFF-LINE. Thus, a subsequent attempt by any host CPU 
to write to the destination storage device 1 10 will be committed (per the step 608 of the 
routine 600). Therefore, the above-described situation wherein application programs on the 
host CPU(s) 102 were caused to crash when a source storage system 104 that lost mirroring 
communication-for a destination storage device 110 simply prevented write-requesting 
command chains from committing is avoided. 

If (at the step 610) it is determined that the destination storage device 1 10 is not 
PENDING OFF-LINE, then the routine 600 recognizes that the destination storage device 1 10 
is currently ON-LINE. In the example shown, when (at the step 610) it is determined that the 
destination storage device 1 10 is ON-LINE, the routine 600 proceeds to. step 614, wherein it is 
determined whether mirroring communication is currently enabled for die destination storage 



WO 01/16748 PCT/US00/24128 

-24- 

device 110. The determination in the step 614 may be made, for example, by checking the 
mirroring link status flags in the table 402 associated with the mirroring group 108 in which 
the destination storage device 1 10 is included to determine whether at least one mirroring link 
1 12 is operational for the mirroring group 108. If one or more of the mirroring links 1 12 for 
the mirroring group 108 are enabled, then it is determined that mirroring communication for 
the destination storage device 1 10 is enabled. Otherwise, it is determined that mirroring 
communication for the destination storage device 1 10 is disabled. 

If it is determined (at the step 614) that mirroring communication is disabled for the 
destination storage device 1 10, the routine 600 proceeds to steps 620-628, wherein actions 
(discussed below) are taken to maintain the consistency of the data set stored by the target 
storage system 106. 

If it is determined (at the step 614) that mirroring communication for the destination 
storage device 1 10 is enabled, then the routine 600 proceeds to the step 616, wherein the 
write-requesting command chain is (conditionally) processed and committed by the target 
storage system 106. The routine 600 then proceeds to step 618, wherein a determination is 
made as to whether mirroring communication remained enabled throughout the processing 
and committal of the write-requesting command chain (i.e., until after the data has been 
transferred successfully to both the source and target storage systems 104 and 106). When it is 
determined that mirroring communication did remain enabled so that the write-requesting 
command chain was permitted to successfully commit, the routine 600 terminates. If, 
however, it is determined (at the step 618) that mirroring communication for the destination 
storage device 110 became disabled before the write-requesting command chain was 
committed in step 616 (e.g., during the transfer of data across the mirroring links 112), then 
the routine 600 proceeds to the step 620, as if the initial check of the table 402 had indicated 
that mirroring communication was disabled for the destination storage device 1 10. The 
determination in step 618 may be made, for example, by monitoring status of the links 
directly, i.e., without reference to the tables 402 (Fig. 4). Therefore, in this embodiment, the 
enablement of mirroring communication for the source storage devices 1 10 is not determined 
solely by reference to the tables 402. Instead, the devices responsible for the mirroring 
communication (e.g., the mirroring controller 308 of Fig. 3), or some other device, can also 
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make a determination that mirroring communication for a source storage device 1 1 0 has 
become disabled. 

When the routine 600 reaches the steps 620-622, it has determined that the source 
storage device 1 10 to which the write-requesting command chain (that resulted in the routine 

5 600 being called) was directed is ON-LINE and cannot engage in mirroring communication 
with the target storage system 1 06. Because the destination storage device 1 1 0 is still ON- 
LINE, the routine 600 recognizes that the write-requesting command chain being processed is 
the first write-requesting command chain directed to a source storage device 1 1 0 in the 
destination storage device's mirroring group 108 since mirroring communication for the 

10 mirroring group 108 became disabled. Therefore ; in the illustrative embodiment shown, the 
routine 600 takes steps to alter the states of all of the other source storage devices 1 10 in the 
destination storage device's mirroring group 108 to either OFF-LINE (step 620) or PENDING 
OFF-LINE (step 622) to account for the loss of mirroring communication for the mirroring 
group 108. 

15 More specifically, at the step 620, for each of the source storage devices 1 10 that 

shares mirroring communication with the destination storage device 1 10 (i.e., that is in the 
destination storage devices mirroring group 10S), and that is not a member of a consistency 
group which may be determined, for example, by checking the consistency group flag in table 
504 of Fig. 5 for the destination storage device 110, the routine 600 sets the state flag for that 

20 source storage device 1 10 to OFF-LINE. Thus, upon subsequent iterations of the routine 600 
involving write-requesting command chains directed to these source storage devices 1 10 that 
have been turned OFF-LINE, the wTite-requesting command chains will be permitted to 
process and commit per steps 608 and 606 of the routine 600, despite the fact that mirroring 
communication has been disabled for those source storage devices 110. 

25 Similarly, at the step 622, for each of the source storage devices 1 1 0 that shares 

mirroring communication with the destination storage device 1 10, and that is a member of a 
consistency group, the routine 600 sets the state flag for that source storage device 1 10 to 
PENDING OFF-LINE. Thus, upon subsequent iterations of the routine 60O involving write- 
requesting command chains directed to these source storage devices 1 10 that have been turned 

30 to the PENDING OFF-LINE state, the routine 600 will proceed to the step 612, wherein the 



10 



WO 01/16748 - 26 " PCT/US00/24128 

speci al I/O failure m essa g e (d.scussed above) will be sent to the host CPU 102 that .ssued the 
I/O command chain. 

After completing .he step 622 of the routine 600, it is determined at step 624 (e.g., by 
chewing the consistency group flag in tabic 504 of Fig. 5) whether the destination storage 

destination storage dev.ce , i 0 is no, a member of a consistency group, then *~ 
pr0 ceeds to the step 626, wherein the state flag for the destinat.on storage devtc 0 
OFF-LINE It is safe to change the state flag for the destination storage dev,ce . 
immediate* to the OFF-LINE state in this situation because the destination storage devtce 

no, a member of a consistency g roup, and the data rumored by the targe, storage 
sys ,em , 06 for this source storage device 1.0 need no, be ice, conststen, w,. the^a 
lored by ,he target storage system 106 for any other source storage devtces 1 10 W 
mutine 600 then proceeds to the step 606, wherein the write-reouesttng command cha n ^ 

Iina.es As mentioned above, when the state flag for a source storage dev.ce 110 is ,n the 
« state, the source storage system ,0, mcluding that storage dev.ce 1 10 is perm, ed 

he transferred to the target storage system 106. 

I is de,erm,ned (a, ,he step 624, tha, ,he delation storage device ■ , 0 ,s a member 
0 of a constancy group, ,hen ,he routine 600 proceeds to ,he s,ep 62 8 , where.n ,he s«,e flag 
2 l,,naHs,oragedevicen0isse,,oPEN D ,NGOFF-LrNE. As dtscussed above, 
: a destination storage device , , 0 ,s in ,he PENOIKO OFF-LTNE s,a,e, ,he source storage 
sy s,em .04 including ,he destination storage device U0 is temporaniy prevented from 
om^in g -wr„=-re q ues,in g corramand chains directed ,o ,he destination s.orage dev.ce 0. 
25 Z is also dtscussed above, according .o one aspec. of the invention, this preventing of the 
source s.orage system 104 from committing such write-reo.uest.ng command Cams ,s 
performed only un.i. mirroring communication for a., of .he o.her source storage evtces 110 
Uded ,n the same consistency group as the destination storage device 1 .0 has been 

30 diSabled ,n the emood.men. of F.g. 6, responsible for disabling m.rroring communication for 
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the source storage devices 1 1 0 that are included in mirroring groups 108 other than the 
disabled mirroring group .08 that includes the destination storage device 1 10 is delegated to 
the host CPU(s) 102. In the illustrative method of Fig. 6, this goal is achieved at the step 612, 
wherein a specia! I/O failure message is transmitted to the host CPU 102 that issued the wnte- 
5 requesting command chain directed to the destination storage device 110. 

As is explained in more detail below, in the illustrative embodiment of F,g. 7, m 
response to receiving this specia, I/O failure message, the hos, CPU 102 sends commands to 
each of the source storage systems 104 including source storage devices 110 included m .he 
same consistency group as the destination storage device 1 1 0. These commands instruct the 
10 source storage systems 104 to se, the state flags for the source storage devices , ,0 included ,n 
the relevant consistency group to .he PENDING OFF-LINE state. After a„ such source 
storage devices 110 have been placed in the PENDING OFF-LINE su.e, the host CPU .02 
tha, recetved the special I/O failure message then sends commands to .he relevant source 
storage systems 104 instructing them to se, the state flags for the source storage dev.ces 1 1 
,5 in Jrelevan, consistency group to the OFF-LINE s.a.e. Finally, after all source storage 
devices 1 10 in the relevant consistency group have been placed in .he OFF-LINE state, the 
source storage systems 104 including these source s.orage devices 1 ,0 are again perm.tted to 
comm., write-requesting command chains directed thereto. Thus, using .his technique, the 
software running on the host CPU(s) 102 tha, invokes me write-reques, command cham w.ll 
20 no, crash because of repea,ed faUures ,0 write ,0 source storage devices . .0 included m a 
disabled mirroring group .08. Rather, whenever a host CPU 1 02 attempts ,o write to a 
destination storage device 110 in . disabled mirroring group 108, me source storage system 
, 04 including the destination storage devtce 1 10 wil, return the special I/O failure message to 
thehos, GPU 102, .hereby instruc.ing ,he host CPU .02 ,o take the above-discussed acuons to 
25 disable mirroring communication for .he source s.orage devices 1,0 in the destination s.orage 
device^ constancy group. After tak.ng such actions, the host CPU 102 will .hen be 
permitted .o wri.e successfully to the destination s.orage device 1 10, even though mirronng 
communication for the destination storage device ,1 0 has become disabled. 

With regard to the above-descnbed steps 620-622, wherein the states of all source 
30 stora-e devices". 10 in the disabled mirroring group 108 are altered, i, should be understood 
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fta. the states of these source storage devices U0 could alternatively be a.tered in other ways, 
and need not be altered by the source storage system 104 during the ,era,io„ of the routtne 
600 of Fig. 6 in which the loss of mirroring communication was discovered. For example, 
fteir states could be altered by the source storage system 104 during subseouen, iterations of 
ft. routine 600, or couid be altered by one of the ho, CPU(s) 1 02 during execut.on of the 
routine 700 of Fig. 7 (discussed below). 

I. this respect, i, should be appreciated that, in the illustrative embodtmen, of F,g. 6, 
ft. states of a« of the storage source devices 1 10 ,n the destination storage devices'* 
mirroring group 108 are altered in the steps 620-622 only as a matter of convcn.enc. ,, 
Tthose source storage devices , 10 in the disabled mtrroring group 10S tha, are m the same 
consistency, group as the desnnation storage device 1 1 0, aitering ft. states of sue source 
stot age devices 1 ,0 to the PENDING OFF-LINE state during the firs, ttera.ton of the routtne 
« m which the loss of —cation is Covered reduces the total number of state flags 
mat the host CPU(s) .02 wii, have to alter to the PENDING OFF-LINE state when 

sys em ,04 in the step 6,2 after the state flag for the destination storage dev.ee s set to ft. 
PENDING OFF-LINE state in the step 62S. In addition, for the source storage devces 1 , 0 n 
" disabled mirroring group ,08 that ar. included in consistency groups other ftan fta - 
destination storage device ,00, or that ar. no, included in any consistency V™ *™^ 
0 states of these source storage devices 1,0 to the PENDING OFF-LINE state or ft. OFF-LINE 
Z respectively, prevents later iterations of the rou,,ne 600 from having .o perform the step 
6,4-628 when a hostCPU ,02 attempts to wr.«e to one of these source storage dev.ces , ,0 or 
wh ich rn.rror.ng communication has aiready been detained to be disabled. In th.s respect, 

25 has aiready been determined to be disabled. In this respect, after the states of al, of the source 
.orage devices ,10 in the disabled mirronng group 10S have been altered, when a wr„e . 
attempted to one of these source storage devices , ,0, the rout.ne 600 wiil never reach the step 
M4because each will have been previously placed in either the OFF-LINE or the PENDING 

30 " Whi,e, .n the embodiment of Fig. 6, the routine 600 causes the states of source storage 
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devices 110 ,0 be altered only in response to a write-requesting command chain being directed 
t0 a destination storage device 110 included in a disabled mirroring group 108, ,t should be 
appreciated that the present invention is not so limited, and tha, the states of the source 
storage devices 1 10 may be alternatively be altered tndependen, of received write-requesttng 
5 command chains. For example, one or more of the controllers in the source storage system 
104 (e g controllers 302, 304, and 308 of Fig. 3) may continuously monitor the status of the 
mirroring links 1 12a-b, either directly or by means of link status tables (e.g., tables 402A-B of 
Fig 4) and these controller^, may alter the state of each of the source storage devices 110 m 
a mirroring group 108 whenever i, is determined tha, mirronng communication has been lost 

10 for that mirroring group 108. ,„.,.„, 
Fig 7 shows an example of a routine 700 tha. may be performed by a host CPU 
whenever the host CPU ,02 receives the special „C failure message from a 
system 104 as discussed above. The routine 700 can be imp.emen.ed on the host CPU 102 
any of numerous ways so that it is responsive to the special I/O faUure message, and the 
,5 invention is no, limited to any particular implementation technique. For example, when the 
host CPU(S) 102 are implemented using mamframe computers, the routine 700 may, for 
example, be embeddedin the code for the device drivers responsible for .ssuing wrtte- 
requesting command chains ,0 the source storage systems ,04. ,„ such an embodiment, the 
specia, error message returned to the host CPU(s) ,02 will be received by the devtce dnver, 
2 0 which can execute this embedded code to perform the routine 700. Alternatively, when the 
host CPU(s) ,02 are implemented using open system computers, the routine 700 may for 
example, be impiemented at a higher ,eve, than the device driver, and the receipt of t e speca, 
„0 fai,ure message from a source storage system ,04 may cause a process.ng thread to 
™ from the device driver code to separately implemented code for the routine 700, ,.e„ 
25 special l/O fai.ure message may temporarily "steal the processing kerne," from the devtce 
driver ,. shou.d be appreciated that the code for the routine 700 cou,d also be embedded m 
,he device driver code for an open system computer. However, it is often the case that the 
device driver code for open systems is proprietary, and is therefore difficult to access and 
mod ify Numerous other suitabie techniques tha, achieve a similar result wi„ readtly occur to 
30 those skilled in the art, and the invention is not limited to the specific examples descnbed. 



WO 01/16748 - 30 " PCT/US00/24128 

With re-ard to the illustrative routine 700 of Fig. 7, it should be appreciated that the 
precise order of the method steps ,s not critical, and that the invention is no, limited to 
embod.ments tha, perform method steps precise* in the order shown. Additionally, should 
be appreciated tha. the method steps shown in Fig. 7 represent on,y one of numerous possible 
routines tha, can achieve the desired result, and the invention is no, Hmi.ed ,o the particular 
routine shown. Further, i, should be understood tha, some embodtments of the inventton can 
perform fewer than a,l of the functions performed by the method steps illustrated in F,g. 7 
and the invent.on is no, hmited to embodiments that emp,oy all of the functions performed by 
the illustrated routine. 

, The routine 700 is called when the special I/O failure message is received from 

source storage system ,04. When the special I/O failure message is received, the routine 700 
proceeds to step 704, wherein ,h= routine 700 identifies all of ,he source storage devtces 1 0 
1 are tncluded ,n the same consistency group as the destination storage device U0 to wh.ch 
the ^e-requesting command chain ,ha, prompted the spec.a. I/O failure message was 
,5 directed. To identify the source storage device U0 to wh.ch the wnte-requesting command 
chain that prompted the specia, l/O failure message was directed, the host CPU(s, can e.ther 
rel y on its own memory regardmg the source storage device to which it sen, the wrtte 
requesting command chatn, or can re,y on informal embedded in ,he speca, I/O fatlure 
message itself tdentifying the source storage device 1 . 0 to which the write-requesttng 
,„ command chain was directed." ,n the embodiment shown, the host CPU(s, ,02 — 
" information regarding which source storage devices are members of which consistency 

gro ups Thus, the function of the step 704 can be performed by the host CPU(s) ,02 s,mp,y 
by refernng to this information for the source storage device ,10 to which the wrtte- 
requesting command chain tha. prompted the specia. I/O failure message was directed. After 
,5 completing the step 704, the routine 700 proceeds to step 706. 

A, the step 706, the routine 700 determines whether there are any source storage 
devices 1 .0 in the relevant constancy group tha. are m the ON-LINE state, and if so, 
proceeds to step 708. At the step 708, for each source storage device 1 , 0 identified ,„ the step 
706 as being in the ON-LINE state, the routine 700 causes the host CPU 102 to issue an I/O 
30 command c'hain mstructing the source storage system ,04 including o» source storage dev.ee 
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1 1 0 to turn it to the PENDING OFF-LINE state. 

The results of steps 706 and 708 may be achieved in any of a number of ways, and the 
invention is no, limited to any particular technique for accomplishing the same- The host 
CPU 107 may, for example, sequentially poll each of the relevant source storage dev.cs 1 10 
to determine its current state, and, if any polled source storage device 1 1 0 remams to be 
turned to the PENDING OFF-LINE state, the host CPU 1 02 may issue an I/O command cha.n 
,„ effect a change of that device's state. To avoid polling devices that have already been 
turned to the PENDING OFF-LINE sta.e, before the host CPU 102 begins perfomnng the 
steps 706 and 708, it may, for example, firs, request an update to a status table (e.g., stored m 
th e hos, CPU's memory) for ,he relevant source s,ora g e devices ! 10 ,n which the current state 
ofeach device is indicated. The host CPU 1 02 could then poll only the relevant source 
storage devices 1 1 0 tha, the table ind.cates have no, ye, been turned ,o ,he PENDING OFF- 
LINE state.and issue command chains ,o change ,„e state of those devices .hat the po mg 
i„dica,es have no, ye, been changed. This technique may save ,ime when multiple host CPUs 
, 0 7 are performing the routine 700 simultaneously, as discussed above. Alternately, each 
host CPU 1 02 performs the routine 700 may, for example, simply issue an I/O command 
chain for each source storage device 1.0 included ,n the destination storage dev.ce s 
constancy group, without updating a table or doing any polling of current device states, an 
let the source storage systems ,04 determine whether or no, ,0 disregard I/O command chatns 
requesting state changes that have already been effected. 

, f i, is determined (a, the step 706) tha. all of the source storage devices 1 10 ,n the 
destination s.orage device's cons.stency group have been turned .0 .he PENDING OFF-LINE 
state the routine 700 proceeds to step 710. At the step 7,0, the routine 700 determ.nes 
^el.. * e ,e are anv source storage devices ,10 m the relevant consistency group that have 
; „o, ye. been turned to the OFF-LINE state, and if so, the routine 700 proceeds to the step 712. 

At the step 712. for each source s.orage device 1 10 identified in the step 710 as bemg 
in the OFF-LINE state, the routine 700 causes the hos, CPU 102 to issue an I/O command 
chain ins,ruc,ing the source storage system 104 including tha, source storage device 1 10 to 
,um i, to the OFF-LINE state. Techniques similar to those described above in connection 
0 with the steps 706 and 708 can be employed to achieve the results of the steps 710 and 712. 
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When i, is determined (at the step 710) that a.l of the source storage devices 1 10 in the 
des.ina.ion storage device's consistency group have been turned to the OFF-LINE state, the 

routine 700 terminates. 

,n tine embodiment shown in Fig. 7, the source storage systems 104 store mformahon 

identifying only whether or „o, each source storage devtce 11 0 is included in a consistency 

group and the host CPU(s) 102 store all information regarding which source storage devtces 
,0 are included in which consistency groups. Thus, in the illustrative embodiment shown, 

one of the host CPUs 102 ts the entity that sends commands requesting that the source storage 

ice! HO in the relevant consistency group he placed first in the PENDING OFF-LINE 
state and then in the OFF-LINE state. I. should be appreciated, however, that the disablmg of 
mirroring communication for the source storage devices ! 1 0 in the relevant consistency group 
can be performed instead by one or more of the source storage systems 104, or any other 
dev.ce or devtces in a computer system that is in communication with the source storage 

^'^embodiments of the present mven.ion described above are directed to a data 
' mirroring system including a, leas, one host, a, leas, one source storage system, and a. leas, 
one J. storage system, wherein the host performs write operations to the source storag 
system and wherein mirroring communication is established between the source and targe, 
slage' sysrems. In .he particular embodiments described above, the host is descnbed as a 
0 host CPU. However, i, should be appreciated .ha. ,he various embodiments of the presen. 
,„ven,ion discussed above are no, hmited in this respect, and can be employed m a data 
mtrtoring sys.em .nduding numerous other ,ypes of host devices capabie of wrmng data to 

the source storage system. 

Having-described several embodiments of the invention in detail, vanous 
25 modifications and improvements will readily occur to .hose skilled in .he ar, Such 
modifications and improvement are in.ended to be within the spin, and scope of the 
inV e„„on. Accordingly, the foregoing description is by way of exampie only, and ,s no. 
in .e„ded as limi.ing. The mven.ion is limi.ed only as defined by .he following cla,ms and 

equivalents thereto. 
30 What is claimed is: 
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1 A method for use in a data mirroring system comprising at least one host, at 
least one source storage system, and at least one target storage system, wherein the at least 
one source storage system includes at least first and second source storage devices, and the at 
least one target storage system includes at least first and second target storage devices, 
wherein the at least one host is coupled to the at least one source storage system to perform 
write operations to the first and second source storage devices, and wherem the at least one 
source storage system is coupled to the at least one target storage system to enable rmrronng 
communication from the first source storage device to the first target storage device and 
from the second source storage device to the second target storage device, the method 

comprising a step of: 

(A) when mirroring communication from the first source storage device to the first 
target storage device is disabled, disabling mirroring communication from the second source 
storage device to the second target storage device. 

2 The method of claim. 1 , wherein the step (A) includes a step of: 

(Al) disabling mirroring communication from the second source storage device to the 

second target storage device in response to a write-requesting command chain directed from 

the at least one host to the first source storage device. 

3 The method of claim 2, wherein the step (Al) includes a step of: 
(A2) in response to the write-requesting command chain directed from the at least one 
ho * ,o *e first source storage device when mirroring communication from the first source 
sto rage device to the first target storage device is disabled, transmitting a first message from 
the at least one source storage system to the at least one host indicating that rmrronng 
communication is disabled from the first source storage device to the first target storage 

device. 

3 4 The method of claim 3, wherein the step (Al) further includes a step of: 

(A3) in response to the at least one host receiving the first message from the at least 
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one source storage system md.atmg that mirroring communication is disabled from the to* 
sourC e storage device to the first target storage dev.ce, transmitting a second message from the 
at le ast one host to the at least one source storage system instructing the at least one source 
lage system to disable mirroring communication from the second source storage de.ce to 
the second target storage device. 

5 The method of claim 1 , wherein the at leas, one source storage system includes 
a p,urali,y of source storage devices in addition to me a, ieast firs, and second source storage 
a! L wherein the a. leas, one target storage system inciudes a piuraiity of target storage 
Z es'in addttion to the a, ,eas, first and second targe, storage devices, wheretn ,he a, le. 
one host ,s coupled ,o the at leas, one source storage system to perform wrtte operat.ons , ,he 
Z * of souL storage devices, wherein the a, leas, one source storage system ,. co 
TL a .east one targe, storage system to enahle mirroring commun.ca.ion from each of he 
l lity of source storage devtces to a corresponding one of ,he pluralhy o target s,ora 
device wherein ,he a, .eas, firs, and second source storage devices are mcluded t a firs, 
, Z f source s,orage devices ,ha, store logically reia,ed da. for which a cons.s.e. 
r P resen,a,ion is desired ,o he maintained on the at ,eas, one target storage system, and 

„ the firs «ar e, storage devtce, disabling mtrroring com—ion from each of ,he 
corresponding one of the plurality of large, storage dev.ces. 

6 ^ method of claim 5, wherein ,he s.ep (Al) includes a «p of maintatning 
. . ' ommumca „on from each of the plurality of source storage devtces that ts not 

of target storage devices. 

7 The method of claim 5, further including a step of: 

(B ) after m.rronng —Con is tabled from the first source storage dev.ee to 
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to ,he f,rs, source storage device from — ing until after mirroring 

been disab,ed from all of the plurality of source storage devices that are tnc.uded mthe first 

g rou p of source storage devices .o the corresponding ones of the p.uraltty of target storage 

devices. 

8 The method of claim 5, further including a step of 

(B ) when mirroring communication is disabled from any one of the plurality of source 
storage devices inc.uded in first group of source storage devices to the corresponding one of 
pLah.y of target storage devices, preventing command chains from the a, ,east one host 

of target storage devices. 

o The me ,hod of claim 8, wherein the step (B) includes steps of: 
■mi, for each one of the plurality of source storage devices included in the firs, group 
of sour e storage devices, storing information in the ftrs, source storage system indicattng 
" sponse to a write-retmestn, command cha.n directed to the one of the plurahty of 
storage dev.ces, the a, leas, one source storage system should temporally no. comm., 

the command chain; and ' 

<B2) after completing the step (Bl), for each one of the plurality of source storage 
de vices included in the ftrs, group of source stora g e devices, stonng information ,n the first 
source storage system indicattng that, ,n response to a write-reouesting command Cam 
II to , he one of the p.urality of source s,ora g e devices, the a, leas, one source stora 
5 X should commtt the wrt,e-re q uest,n g command chatn without transfers data wrttten 
. thereby to the corresponding one of the plurality of target storage dev.ee, 

10 The method of claim 9, wherein: 

the step (Bl ) includes a step of, for each one of the plurality of source storage dev.ces 
30 included in the firs, group of source storage dev.ces, storing .nformation in the at leas, one 
source storage system indicating ma,, in response ,o a wr.te-reouesting —6 chatn 
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directed ,o the one of the plurality of source storage devices, the a, least one source storage 
system should transnu. a firs, message to the a, leas, one host indicating that m.rronng 
communication is disabled from the one of the plurality of source s,orage devices ,0 the 
corresponding one of the plurality of target storage devices; and 

the step (A.) inc.udes a step of, in response to the at least one host rece.v.ng a firs, 
message from the a, leas, one source storage system indicating tha, mirroring communication 
is disabled from one of the plurality of source storage devices included in the firs, group of 
source storage dev.cesto the corresponding one of the pluraH.y of targe, storage devices, 
transmitting at leas, one second message from ,he a, leas, one host to the at .east one source 
storage system instructing me at ,=ast one source storage system to perform the steps (Bl) and 



(B2). 



] 1 The method of claim 1 , further including a step of: 

(B) after mirroring cornmunica.ion has been disab.ed from the first source storage 
device to the firs, targe, storage device, preventing command chains from the a, ieast one hos, 
directed ,o ,he firs, source s,orage device from committing un,U after mirronng 



storage device. 
12 



1Z The method of claim 1 , wherein the at least one source storage system is 
coupled to the a, leas, one ,ar g e, s.orage sys.em via a, leas, one firs, mirroring link tha, 
enables mirroring cornmunica.ion from the firs, source storage device ,0 the firs, target 
storage device and via a, leas. on. second mirronng link .hat enab.es mirronng 
c _ca,ion from the second source s.orage dev.ce to the second targe, storage dev.ce, and 
wherein ,he s,ep (A) includes a s,ep of disabhng m.rroring communica,ion over the a. least 
one second mirronng link from .he second source s,ora g e device ,o .he second rarget s,orage 
device whi.e perm.mng mirronng communicanon to continue over the a, !east one second 
mirroring link from a ,hird source s.orage device ,o a third target storage dev.ce. 

13 The method of claim 1 , wherein the at least one source storage system is 
coupled to the at leas, one targe, s.orage system via at .east one firs, m.rroring .ink that 
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enables mirroring communication from the first source storage system to the first target 
storage device but that does not enable mirroring communication from the second source 
storage device to the second target storage device, and via at least one second mirroring link 
that enables mirroring communication from the second source storage device to the second 
target storage device but that does not enable mirroring communication from the first source 
storage device to the first target- storage device. 

14 The method of claim 8, wherein the step (B) includes steps of: 
CB1) temporarily placing each of the plurality of source storage devices included in the 
first .roup of source storage devices in a first state in which, in response to a write-requesting 
command chain directed to one of the plurality of source storage devices in the first state, the 
at least one source storage system temporarily does not commit the command chain; and 

(B2) after completing the step (Bl), placing each of the plurality of source storage 
devices included in the first group of source storage devices in a second state in which, m 
response to a write-requesting command chain directed to one of the plurality of source 
storage devices in the second state, the at least one source storage system commits the write- 
requesting command chain without transferring data written thereby to the corresponding one 
of the plurality of target storage devices. 

, 15 The method of claim 14, wherein: 

the step (Bl) includes a step of, when each of the plurality of source storage devices 
included in the first group of source storage devices is in the first state, in response to a write- 
requesting command chain directed to one of the plurality of source storage devices ,n the first 
State the at least- one source storage system transmits a first message to the at least one host 
5 indicating that mirroring communication is disabled from the oneof the plurality of source 
storage devices to the corresponding one of the plurality of target storage devices; and 

^ the step (Al) includes a step of, in response to the at least one host receiving a first 
message from the at least one source storage system indicating that minoring communion 
is disabled from one of the plurality of source storage devices included in the first group of 
50 source storage devices to the corresponding one of the plurality of target storage devices, 

transmitting at least one second message from the at least one host to the at least one source 
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storage system instructing the at least one source storage system to perform the steps (B 1 ) and 
(B2). 

16. A source storage system for use in a data mirroring system including at least 
5 one host and at least one target storage system including at least first and second target storage 
devices of the source storage system, comprising: 

at least first and second source storage devices to be coupled to the at least one host to 
enable the at least one host to perform write operations to the first and second source storage 
devices and further to be coupled to the at least first and second target storage devices, 
10 respectively, to enable mirroring communication from the at least first and second source 
storage devices to the at least first and second target storage devices; and 

" at least one controller to, responsive to mirroring communication from the first source 
storage device to the first target storage device becoming disabled, disable mirroring 
communication from the second source storage device to the second target storage device. 
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1 7 The source storage system of claim 1 6, wherein the at least one controller is 
configured to disable mirroring communication from the second source storage device to the 
second target storage device in response to a write-requesting command chain directed from 
the at least one host to the first source storage device. 

1 8 The source storage system of claim 16, wherein the at least one target storage 
system includes a plurality of target storage devices in addition to the at least first and second 

target storage devices, and wherein: 

the source storaae system includes a plurality of source storage devices in addit.cn to 
the at least first and second source storage devices to be coupled to the at least one host to 
enable the at least one host to perform write operations to the plurality of source storage 
devices and further to be coupled to the plurality of target storage devices to enable rmrronng 
communication from each of the plurality of source storage devices to a corresponding one of 
the plurality of target storage devices, and the at least first and second source storage devices 
are included in a first group of source storage devices that store logically related data for 
which a consistent representation is des.red to be maintained on the at least one target storage 
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system; and 

the at least one controller is configured to, when mirroring communication is disabled 
from the first source storage device to the first target storage device, disable mirroring 
communication from each of the plurality of source storage devices that is included in the first 
group of source storage devices to the corresponding one of the plurality of target storage 
devices. 

,9 The source storage system of claim 18, wherein the at least one controller is 
confined to disable mirroring communication from each of the plurality of source storage 
devices that is included in the first group of source storage devices .6 the correspondmg one 
of the plurality of target storage devices when mirroring communication is disabled from the 
firs, source storage device to the firs, target storage device while maintaining nurrormg 
communication from each of the plurahty of source storage devices that is no. included m the 
firs, group of source storage devices ,0 the corresponding one of the plurality of urge, storage 
devices. 

,0 The source storage system of claim 18, wherein tire at least one controller is 
configured to, after mirronng communication is disabled from the firs, source storage dev.ee 
to the firs, .arget storage device, prevent command chains from the at leas, one host direced 
,o ,he firs, source s,orage device from commimng until after mirroring communicauon has 
been disabled from all of the plurality of source storage devices that are included m the first 
group of source storage devices to the corresponding ones of the plurality of target storage 
devices. 

21 The source storage system of claim 1 8, wherein the at least one con.roller is 
configured ,o, when mirroring communication is d.sabled from any one of .he plurality of 
source storaae devices included in firs, group of source storage devices ,0 the corresponding 
one of ,he plummy of <arge. storage devices, prevent command chains from the at leas, one 
host directed to the one of ,he plurality of source storage devices from committing un.,1 after 
0 mirroring communication has been disabled from all of the plurality of source storage dev.ces 
included in ,he firs, group of source s.orage devices ,o ,he corresponding ones of ,he plummy 
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of tareet storage devices. " _ 

~r, The source storage system of claim 1 6, wherein the at least one controller IS 
confined to, after mirroring communication has been disabled from the firs, source storage 
devic'e to the firs, targe, storage device, prevent command chains from the a, leas, one host 
i directed to the firs, source storage device from committing until after rmrrormg 

communication has been disabled from the second source storage device ,o the second targe, 

storage device. 

,3 The source s,orage sys,em of claim 16, wherein the at least one source storage 
system is to be coupled ,0 ,he a, !eas, one target storage system via a, Las, one firs, mirronng 
0 link tha, enables mirroring communicafion from the firs, source storage device ,o ,he firs, 
target storage device and via a. leas, one second mirronng link that enables mirronng 
colunication from the second source s.orage device ,0 ,he second .arget storage dev.ce, and 
wherein the at leas, one confer is configured ,0 disable mirroring commun,ca«,on over the 
a, leas, one second mirroring link from the second source s,orage device to the second target 
! s storage device while permitting mirroring communication to continue over the a, leas, one 
second mirroring .ink from a third source storage device to a ,h,rd targe, s,orage dev.ce. 

24 The source storage system of claim 1 6, wherein the a. leas, one controller 
includes means for, when mirroring communication from the firs, source s,ora g e device ,o the 
,0 firs, .arge, storage device is disabled, disabling mirroring communication from the second 
source storage device ,o me second large, storage device. 

25 At least one computer-readable medium for use with a, least one processor 
Eluded in a data mirroring system comprising a, .eas, one hos,, a, leas, one source s,orage 
25 svstem, and a, leas, one targe, s,orage system, wherein the a, .eas, one source storage system 
includes a, least firs, and second source storage devices, and the at leas, one ,arge, s,orag= 
sys,em includes a, leas, firs, and second ,arge, storage devices, wherein ,he a, leas, one hos, ,s 
coupled ,o ,he a, leas, one source storage sys,em ,0 perform wri.e opera,ions ,o the firs, and 
second source s.orage devices, and wherein ,he a, .eas, one source s,orage sys,em is coupled 
30 ,o the at leas, one targe, storage sys,em to enable mirroring communication from the firs, 
source storage device to the firs, targe, storage device and from me second source s.orage 
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device to the second target storage device, the at least one computer-readable medium having 
plurality of instructions stored thereon which, when executed by the at least one processor, 
the at least one processor to perform a method including a step of: 
(A) when mirroring communication from the first source storage device to the first 
target storage device is disabled, disabling mirroring communication from the second source 
storage device to the second target storage device. 



a 

cause 



26. The at 
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least one computer-readable medium of claim 25, wherein the step (A) 

includes a step of: 

(Al) disabling mirroring communication from the second source storage dev lC e to the 
• second target storage device in response to a write-requesting command chain directed from 
the at least one host to the first source storage device. 

27. The at least one computer-readable medium of claim 26, wherein the step (Al) 

1 5 includes a step of: 

(A2) in response to the write-requesting command chain directed from the at least one 

host to the first source storage device when mirroring communication from the first source 
storaae device to the first target storage device is disabled, transmitting a first message from 
the at least one source storage system to the at least one host indicating that mirroring 
20 communication is disabled from the first source storage device to the first target storage 
device. 

28. The method of claim 27, wherein the step (Al ) further includes a step of: 
(A3) in response to the at least one host receiving the first message from the at least 
25 one source storage system indicating that mirroring communication is disabled from the first 
source storage device to the first target storage device, transmitting a second message from the 
at least one host to the at least one source storage system instructing the at least one source 
storage system to disable mirroring communication from the second source storage device to 
the second target storage device. 



30 



29. The at least one computer-readable medium of claim 25, wherein the at least 
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one source storage system includes a plurahty of source storage devices in addttion to the at 
L. firs, and second source storage devices, wherein the a, .east one target storage system 
I es a plurality o f target storage devices in add.tion to the at ieas, firs, and second targe, 

" m ,o perform write operations to the pluralny of source storage devices, where.n the a 

l» - » - * - ««* s, ° rage system ,o enab 

laiiv reiated da. for which a consistent represent is desired to he mamtamed on the 
at least one target storage system, and wherein the step (A) includes a step of: 

I,) when mirroring communication ,s disabled from the firs, source storage evtce to 
the firs tar e, storage device, discing mtrroring communication from each of the plura y 
oLrce sLage devices that is included in the firs, group of source storage devrces to the 
corresponding one of the plurality of*** storage devtces. 

30 The at ieas, one computer-readable medium of cla,m 29, where.n the s,ep (Al) 

stora-e devtces tha, ,s not induded ,„ Ore first group of source storage dev.ces to 
j corresponding one of .he plurality of targe, s.orage dev.ces. 

31 . The a, leas, one compu,er.readable medium of claim 29, wheretn ,he method 

further includes a step of. 

, B) .fer mirroring communication ,s disabled from the firs, source storage dev.ee to 
, 5 ,he firsuarge s.orage device, preventing command chains from the a, leas, one bos, directed 
! ,he firs, ource s.orage device from committing until after mirroring comm—n has 
I n dtsabled from all of the piurality of source storage devices that are mciuded m the firs, 
Itp of source storage devices to .he corresponding ones of the p,ura„,y of target storage 

devices. 

30 32 . Th e a, leas, one compu,er-readable medium of claim 29, wherein the method 
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stor Jdevtces .ncluded in first group of source storage devices to the corresponds one o 
IX i, of target storage devices, preventing _d chains from the a, ,east oncost 

, \ „e of the plurality of source storage devices from comm.tt.ng unttl after 
ahected to the one °J » dlsabl ed from aU of the plurality of source storage devices 

of target storage devices. 

a uio n.pHinm of claim 32. wherein the step,(B) 
33. The at least one computer-readable medium ot claim 

' ndUde ;Z°r f each one of the plural.ty of source storage devices included ,n the firs, group 

devices storing information in the firs, source storage system .nd.cat.ng 
ef source ^^ZJ^ _ d chain direcK d to the one of the plurality of 

-1"- 

, indicating that in response to a write-requesting command cham 
sour ce storage ~ <* - ~ ^ ^ 

thereby ,„ the correspondtng one of the plurality of target storage dev.ee, 

34 The a. leas, one computer-readable medium of claim 33. therein: 
he step (B„ includes a step of, for each one of the plurality of source storage ev.es 

X should transmit a first message to the a, leas, one hos, indica,in g tha, m.rror.n 
corresponding one of the plurality of target storage devices; and 



(B2). 

35. 
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the step (A.) incudes a step of, ,n response ,c ,h= at leas, one host receivng a firs, 
I , e a .east one source storage system .ndicating tha. mirroring commumca, o„ 
„ from a U ^ ^ fe ^ fit- group of 

source storage dev.ces ^ ^ ^ tf ^ ^ source 

T.e a, leas, one compu.er-readable medium of Cairn 25, wherein the method 
0 further includes a step of. VMion has been disabled from the first source storage 

communication has been disabled from the second source storage 

1 5 storage device. 

-« The a, least one computer-readable medium of Cairn 25, wherein the a, leas. 

em is coupled .o ,he a. least one targe, storage system via a. leas, one 

firs, m.rronng 1ml .ha. en ^ ^ ^ ^ 

20 ,0 .he first targe. ,orage dev.ce £ ^ ^ ^ ^ ^ ^ 

mirotmg commu : ,^ 

device, and where.n .he s.ep (A) ^ ^ ^ ^ 

37 . Th e a. leas, one computer-readable medium of claim 32, wherein the s.ep (B) 

P-acing each one of .he pluralUy of source storage devices included 
(Bl) tempo yp s(aK , n whkh in respo „ se to a ^ 
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least one source storage system temporarily does not commit the command chain; and 

(B2) after completing the step (Bl), placing each one of the plurality of source storage 
devices included in the first group of source storage devices in a second state in which, in 
response to a write-requesting command chain directed to the one of the plurality of source 
storage devices, the at least one source storage system commits the write-requesting command 
chain without transferring data written thereby to the corresponding one of the plurality of 
target storage devices. 

38. The at least one computer-readable medium of claim 37, wherein: 
the step (Bl) includes a step of, when each one of the plurality of source storage 
devices included in the first group of source storage devices is in the first state, in response to 
a write-requesting command chain directed to the one of the plurality of source storage 
devices the at least one source storage system transmits a first message to the at least one host 
indicating that mirroring communication is disabled from the one of the plurality of source 
storage devices to the corresponding one of the plurality of target storage devices; and 

the step (Al ) includes a step of, in response to the at least one host receiving a first 
message from the at least one source storage system indicating that mirroring communication 
is disabled from one of the plurality of source storage devices included in the first group of 
source storage devices to the corresponding one of the plurality of target storage devices, 
transmitting at least one second message from the at least one host to the at least one source 
storage system instructing the at least one source storage system to perform the steps (Bl) and 
(B2). 

39 A method for use in a data mirroring system comprising at least one host, at 
least one source storage system, and at least one target storage system, wherein the at least 
one source storage system includes at least first and second source storage devices, and the at 
least one tamet storage system includes at least first and second target storage devices, 
wherein the at least one host is coupled to the at least one source storage system to perform 
write operations to the at least first and second source storage devices, and wherein the at least 
0 one source storage system is coupled to the at least one target storage system to enable 

mirroring communication from the first source storage device to the first target storage dev.ce 
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and from the second source storage device to the second target storage device, the method 

comprising a step of: 

(A) in response to the at least one host receiving an indication that mirroring 
communication from the first source storage device to the first target storage device is 
disabled instructing the at least one source storage system to disable mirroring 
communication from the second source storage device to the second target storage device. 

40 The method of claim 39, wherein the at least one source storage system 
includes a plurality of source storage devices in addition to the at least first and second source 
storage devices, wherein the at least one target storage system includes a plurality of target 
storaoe devices in addition to the at least first and second target storage devices, wherem the 
at least one host is coupled to the at least one source storage system to perform write 
operations to the plurality of source storage devices, wherein the at least one source storage 
system is coupled to the at least one target storage system to enable mirroring communion 
from each of the pluraUty of source storage devices to a corresponding one of the plurality of 
target storage devices, wherein the at least first and second source storage devices are 
included in a first group of source storage devices that store logically related data for which a 
consistent representation is desired to be maintained on the at least one target storage system, 
and wherein the step (A) includes a step of: 

(Al) in response to the at least one host receiving the indication that mirroring 
communication from the first source storage device to the first target storage device is 
disabled, instructing the at least one source storage system to disable mirroring 
communication from each of the plurality of source storage devices that ,s included in the first 
gr0 up of source storage devices to the corresponding one of the plurality of target storage 
i devices. 

41 . The method of claim 40, wherein the step (Al) includes steps of: 
(A?) instructing the at least one source storage system to temporarily not commit 
write-requesting command chains directed to ones of the plurality of source storage devices 
0 that are included in the first group of source storage devices; and 

(A3) after completing the step (A2), instructing the at least one source storage system 
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to commit write-requesting command chains directed to ones of the plurality of source storage 
devices that are included in the first group of storage devices without transferring data written 
thereby to corresponding ones of the plurality of target storage devices. 

42. The method of claim 40, wherein the step (Al) includes a step of maintaining 
mirroring communication from each of the plurality of source storage devices not included in 
the first group of source storage devices to the corresponding one of the plurality of target 
storage devices. 

43 . The method of claim 39, wherein the at least one source storage system is 
coupled to the at least one target storage system via at least one first mirroring link that 
enables mirroring communication from the first source storage device to the first target 
storage device and via at least one second mirroring link that enables mirroring 
communication from the second source storage device to the second target storage device, and 
15 wherein the step (A) includes a step of instructing the at least one source storage system to 
disable mirroring communication over the at least one second mirroring link from the second 
source storage device to the second target storage device while permitting mirroring 
communication to continue over the at least one second mirroring link from a third source 
storage device to a third target storage device. 
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44. The method of claim 39, wherein the at least one source storage system is 
coupled to the at least one target storage system via at least one first mirroring link that 
enables mirroring communication. from the first source storage system to the first target 
storage device but that does not enable mirroring communication from the second source 
storage device to the second target storage device, and via at least one second mirroring link 
that enables mirroring communication from the second source storage device to the second 
target storage device but that does not enable mirroring communication from the first source 
storage device to the first target storage device. 

30 45. At least one computer-readable medium for use with at least one host processor 

of a data mirroring system including the at least one host processor, at least one source storage 
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system and a, leas, one target storage system, wherein the at least one source storage system 
includes at leas, first and second source storage devices, and the a. leas, one targe, storage 
system includes a. leas. firs, and second targe, storage devices, wherein the a. least one host 
processor is coupled ,o ,he at leas, one source storage system ,0 perform write operations ,o 
t „e a, leas, firs, and second source s,orage devices, and wherein ,he a, leas, one source s,orage 
sys,em is coupled ,o ,he a, leas, one <arge, storage system to enab!e mirroring communicauon 
from the firs, source storage device ,o ,he firs, ,arge, storage device and from ,he second 
source s,orage device to the second targe, storage device, the a, leas, one compmer-reada ie 
medium having a plurality of ins,ruc,ions s.ored .hereon which, when executed by ,he a, leas, 
one hos, processor, cause ,he a, leas, one processor ,o perform a method including a step of: 

(A) in response to the at least one hos, processor receiving an ind.car.on that 
mirroring communication from ,he first source storage device to ,he firs, ,arge, s,ora g e dev.ee 
is disabled, insuring ,he a, leas, one source storage system to disable m.rronng 
communication from the second source storage device ,o ,he second target storage dev.ee. 

46 The a, leas, one compu,er-readable medium of claim 45, wherein the at leas, 
one source s.orage sys,em includes a plurali.y of source s,orage devices in addition ,0 the a, 
,eas, firs, and second source s.orage devices, wherein ,he a. ieas, one ,arge, storage system 
includes a pluralhy of target storage devices in add.tion to the a. ieas, firs, and second ,arge, 
, s.orage devices, wherein the at least one host is coupled to the at leas, one source s.orage 
sysrem to perform wri.e opera.ions to the plurality of source srorage devices, where.n the X 

mirroring communica.ion from each of the plurali.y of source storage devices to a 
corresponding one of the plurality of ,arge, s,orage devices, wherein the a, .eas, firs, and 
, 5 second source storage devices are .ncluded in a firs, group of source s,ora g e dev.ces ,ha, sure 
' l0K ica„y re,a.=d da. for which a consistent representation ,s desired ,o be ma,n«a,ned on the 
aUeast one targe, storage system, and wherein the step (A) includes a step of: 

(Al) in response to the a. leas, one host processor receiving the indicator, that 
m.rronng communication from the firs, source s.orage device ,o ,he firs, target storage dev.ee 
30 is disabled, ins,ruc,.ng ,he a, leas, one source s.orage sys.em ,o disable mirrormg 

communication from each of .he p.ural.ty of source storage devices that ,s .nCuded ,n the 
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gr0 up of source storage devices ,0 the corresponding one of .he plurali<y of target storage 

devices. 

47. The at least one computer-readable medium of claim 46, wherein the step (Al) 

includes steps of: 

(A2) instructing the at least one source storage system to temporarily not comm.t 
„ri,e-re,ues.ing c«d chains directed to ones of the plurality of source storage dev.ces 
that are included in the first group of source storage devices; and 

(A3) after completing the step (A2), instructing the at leas, one source storage system 
ro commit wri.e-reoues.ing command chains directed to ones of the plurality of source storage 
devices that are included ,n the firs, group of storage devices without transferring data wntten 
thereby to corresponding ones of the plurality of targe, storage dev.ces. 

48 The method of claim 46, wherein the step (Al) includes a s.ep of maintaining 
mirroring commumcation from each of the plurality of source storage devices that ,s no. 
inc ,uded in ,he firs, group of source storage devices ,0 the correspond one of the plural.ty 

of target storage devices. 

49 The method of claim 45, wherein the at least one source storage system is 
coupled to the a, least one target storage system via a. leas, one firs, mirroring In* ,ha, 
enables mirroring communication from the firs, source storage device to the firs, target 
s ,ora»e device and via a, leas, one second mirroring link .ha, enables mirronng 
communication from the second source storage device ,o ,he second ,arge, s.orage dev.ee and 

..he step (A) includes as.ep of ,ns,ruc,in g ,he a, leas, one source storage system «o 
dille mirroring commun.cation over the a, leas, one second mirroring Unk from <he second 
source s.orage device .0 the second ,arge, s,orage dev.ee while perm.mng m.rronng 
communication ,0 conunue over ,he a, leas, one second mirroring link from a .h,rd source 
storage device ,o a third targe, storage device, 

50 A hos, for use in a data m.rroring sys.em including a, leas, one source s.orage 
J sy s,em, and a. leas, one .arge. s.orage sys.em, wherein the a, leas, one source s.orage sys.em 
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includes at least first and second source storage devices to store date written by the host, and 
the at least one target storage system includes at least first and second target storage dev.ces, 
and wherein the at least one source storage system is coupled to the at least one target storage 
system to enable mirroring communication from the first source storage device to the first 
target storage device and from the second source storage device to the second target storage 



device, the host comprising: 

at least one controller to, responsive to receiving an indication that mirroring 
communication from the first source storage device to the first target storage device is 
disabled, instruct the at least one source storage system to disable mirroring communion 
from the second source storage device to the second target storage device. 

5 1 The host of claim 50, wherein the at least one source storage system includes a 
plurality of source storage devices in addition to the at least first and second source storage 
devices wherein the at least one target storage system includes a plurality of target storage 
devices m addition to the at least first and second target storage devices, wherein the at least 
one source storage system is coupled to the at least one target storage system to enable 
mirroring communication from each of the plurality of source storage dev 1C es to a 
corresponding one of the plurality of target storage devices, wherein the at least first and 
second source storage devices are included in a first group of source storage devices that store 
, logically related data for which a consistent representation rs desired to be maintained on the 
at least one target storage system, and wherein: 

the at least one controller is configured to, responsive to receiving the indication that 
mirroring communication from the first source storage dev.ce to the first target storage device 
i, disabled, instruct the at least one source storage system to disable mirroring commumcanon 
5 from each of the plurality of source storage devices that i, included in the first group of source 
storage devices to the corresponding one of the plurality of target storage devrces. 

52. The host of claim 5 1 , wherein the at least one controller is configured to, 
responsive to receiving the indication that mirroring communication from the first source 
50 storage device to the first target storage device is disabled, instruct the at least one source 

storage system to temporarily not commit write-reouesting command chains directed to ones 
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of the plurality of source storage devices that are included in the first group of source storage 
devices, and is further configured to, after instructing the at least one source storage system to 
temporarily not commit write-requesting command chains directed to ones of the plurality of 
source storage devices that are included in the first group of source storage devices, instruct 
the at least one source storage system to commit write-requesting command chains directed to 
ones of the plurality of source storage devices that are included in the first group of storage 
devices without transferring data written thereby to corresponding ones of the plurality of 
target storage devices. 

53. The host of claim 50, wherein the at least one source storage system is coupled 
to the at least one target storage system via at least one first mirroring link that enables 
mirroring communication from the first source storage device to the first target storage dev.ce 
and via at least one second mirroring link that enables mirroring communication from the 
second source storage device to the second target storage device, and wherein the at least one 
controller is configured to, responsive to receiving the indication that mirroring 
communication from the first source storage device to the first target storage device is 
disabled, instruct the at least one source storage system to disable mirroring communicate 
over the at least one second mirroring link from the second source storage device to the 
second target storage device while permitting mirroring communication to continue over the 
at least one second mirroring link from a third source storage device to a third target storage 
device. 

54. A method for use in a data mirroring system comprising at least one host, at least 
one source storage system, and at least one target storage system, wherein the at least one 
source storage system includes a plurality of mirrored source storage devices, and the at least 
one target storage system includes a plurality of target storage devices, wherein the at least 
one host is coupled to the at least one source storage system to perform write operations to the 
plurality of mirrored source storage devices, and wherein the at least one source storage 
system is coupled to the at least one target storage system to enable mirroring communication 
0 from each of the plurality of mirrored source storage devices to a corresponding one of the 
plurality of target storage devices, the method comprising steps of: 
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(A) storing information in the data mirroring system identifying at least one subset of 
the plurality of mirrored source storage devices for which a consistent representation of data is 
desired to be maintained on at least one corresponding subset of the plurality of target storage 



devices. 
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55. The method of claim 54, further including a step of: 
(B) using the information stored in the step (A) to ensure that a consistent 
representation of data is maintained on the at least one corresponding subset of the plurality of 

target storage devices. 

56 The method of claim 54, wherein the at least one subset of the plurality of 
mirrored source storage devices includes a plurality of subsets of the plurality of mirrored 
source storage devices, and wherein the step (A) includes a step of storing information « the 
a, leas, one host identifying which of the plurality of mirrored source storage dev.ces are 
included in which of the plura.i,y of subsets of the plurality of mirrored source storage 
devices. 

57 The method of claim 56, wherein the step (A) further includes a step of: 
CA1) storing information in the a, least one source storage system identifying whether 

each of the plurality of mirrored source storage devices is a member of a consistency group. 

58 The method of claim 54, wherein the step (A) further includes a step of: 
(Al) storing information in the a. least one source storage system identifying whether 

each of the pluraiity of mirrored source storage devices is a member of a consistency group. 

59 A host for use in a data mirroring system comprising a. least one source storage 
system inciuding a plurality of mirrored source storage devices, and a, leas, one target storage 
system including a plurality of target storage dev.ces, wherein ,he a, least one source storage 
system is coupled to the a, leas, one ,arge, storage system ,o enable mirroring comnvumcauon 
from each of ,he plura.ity of mirrored source s,o,age dev.cesto a corresponding one of the 
plurality of target storage devices, the host comprising: 
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at least one storage element; and 

at least one controller to be coupled to the at least one source storage system to 
perform write operations to the plurality of mirrored source storage devices, the at least one 
controller being configured to store information in the at least one storage element identifying 
at least one subset of the plurality of mirrored source storage devices for which a consistent 
representation of data is desired to be maintained on at least one corresponding subset of the 
plurality of target storage devices. 

60. The host of claim 59, wherein the at least one subset of the plurality of 
mirrored source storage devices includes a plurality of subsets of the plurality of mirrored 
source storage devices, and wherein the at least one controller is configured to store 
information in the at least one storage element identifying which of the plurality of mirrored 
source storage devices are included in which of the plurality of subsets of the plurality of 
mirrored source storage devices 

61. A source storage system for use in a data mirroring system comprising at least 
one host, and at least one target storage system including a plurality of target storage devices, 
the source storage system comprising: 

a plurality of mirrored source storage devices to be coupled to the at least one host to 
permit the at least one host to perform write operations to the plurality of mirrored source 
storage devices, and to be further coupled to the at least one target storage system to enable 
mirroring communication from each of the plurality of mirrored source storage devices to a 
corresponding one of the plurality of target storage devices; 
at least one storage element; and 

at least one controller to store information in the at least one storage element 
identifying ones of the plurality of mirrored source storage devices that each belongs to a 
subset of the plurality of mirrored source storage devices for which a consistent representation 
of data is desired to be maintained on at least one corresponding subset of the plurality of 
target storage devices. 
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storage system of claim 61, wherein the at least one controller is 



10 



15 



25 



WO 01/16748 " 54 - PCT/US00/24128 

configured to store the information in the at least one storage element identifying whether 
each of the plurality of mirrored source storage devices is a member of a consistency group. 

63 A method for use in a data mirroring system comprising at least one host, at 
least one source storage system, and at least one target storage system, wherein the at least 
one source storage system includes a plurality of mirrored source storage devices, and the at 
least one target storage system includes a plurality of target storage devices, wherein the at 
least one host is coupled to the at least one source storage system to perform write operations 
to the plurality of mirrored source storage devices, and wherein the at least one source storage 
system is coupled to the at least one target storage system to enable mirroring communication 
from each of the plurality of mirrored source storage devices to a corresponding one of the 
■ plurality of target storage devices, the method comprising steps of: 

(A) placing at least one of the plurality of mirrored source storage devices in one of a 
first state, a second state, and a third state; 

(Bl) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the first state, permitting 
the command chain to commit and transferring data written by the command chain to the 
corresponding one of the plurality of target storage devices; 

(B2) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the second state, 
preventing the command chain from committing; and 

(B3) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of rmrrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the third state, permitting 
the command chain to commit w.thout transferring data written by the command chain to the 
corresponding one of the plurality of target storage devices. 

50 64 A source storage system for use in a data mirroring system including at least 

one host, and at least one target storage system including a plurality of target storage devices, 
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the source storage system comprising: 

a plurality of mirrored source storage devices to be coupled to the at least one host to 
permit the at least one host to perform write operations to the plurality of mirrored source 
storage devices, and to be further coupled to the plurality of target storage devices to enable 
mirroring communication from each of the plurality of mirrored source storage devices to a 
corresponding one of the plurality of target storage devices; and 

at least one controller to place at least one of the plurality of mirrored source storage 
devices in one of a first state, a second state, and a third state, wherein, in response to the 
source storage system receiving a write-requesting command chain directed to the at least one 
of the plurality of mirrored source storage devices when the at least one of the plurality of 
mirrored source storage devices is in the first state, the at least one controller permits the 
command chain to commit and transfers data written by the command chain to the 
corresponding one of the plurality of target storage devices, wherein, in response to the source 
storage system receiving a write-requesting command chain directed to the at least one of the 
plurality of mirrored source storage devices when the at least one of the plurality of mirrored 
source storage devices is in the second state, the at least one controller prevents the command 
chain from committing, and wherein, in response to the source storage system receiving a 
write-requesting command chain directed to the at least one of the plurality of mirrored source 
storage devices when the at least one of the plurality of mirrored source storage devices is in 
the third state, the at least one controller permits the command chain to commit without 
transferring data written by the command chain to the corresponding one of the plurality of 
target storage devices. 

65 At least one computer-readable medium for use with at least one processor 
included in a data mirroring system comprising at least one host, at least one source storage 
system, and at least one target storage system, wherein the at least one source storage system 
includes a plurality of mirrored source storage devices, and the at least one target storage 
system includes a plurality of target storage devices, wherein the at least one host is coupled 
' to the at least one source storage system to perform write operations to the plurality of 
mirrored source storage devices, and wherein the at least one source storage system is coupled 
to the at least one target storage system to enable mirroring communication from each of the 
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plurality of mirrored source storage devices to a corresponding one of the plurality of target 
storage devices, the at least one computer-readable medium having a plurality of instructions 
stored thereon which, when executed by the at least one processor, cause the at least one 
processor to perform a method including steps of: 

(A) placing at least one of the plurality of mirrored source storage devices in one of a 

first state, a second state, and a third state; 

(B 1 ) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the first state, permitting 
the command chain to commit and transferring data written by the command chain to the 
corresponding one of the plurality of target storage devices; 

(B2) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage devices when the 
at least one of the plurality of mirrored source storage devices is in the second state, 
preventing the command chain from committing; and 

(B3) in response to the source storage system receiving a write-requesting command 
chain directed to the at least one of the plurality of mirrored source storage, devices when the 
at least one of the plurality of mirrored source storage devices is in the third state, permitting 
the command chain to commit without transferring data written by the command chain to the 
corresponding one of the plurality of target storage devices. 
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